2023-04-12 02:53:45,271 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure timeout: 13 mins 2023-04-12 02:53:45,542 DEBUG [main] hbase.HBaseTestingUtility(342): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/22e25865-4487-e2aa-7556-0db30ee3c8ff 2023-04-12 02:53:45,557 INFO [Time-limited test] hbase.HBaseTestingUtility(1068): Starting up minicluster with option: StartMiniClusterOption{numMasters=1, masterClass=null, numRegionServers=3, rsPorts=, rsClass=null, numDataNodes=3, dataNodeHosts=null, numZkServers=1, createRootDir=false, createWALDir=false} 2023-04-12 02:53:45,557 INFO [Time-limited test] hbase.HBaseZKTestingUtility(82): Created new mini-cluster data directory: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/22e25865-4487-e2aa-7556-0db30ee3c8ff/cluster_2d9c59b9-d025-040c-6051-62d17e49a4b2, deleteOnExit=true 2023-04-12 02:53:45,558 INFO [Time-limited test] hbase.HBaseTestingUtility(1082): STARTING DFS 2023-04-12 02:53:45,558 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting test.cache.data to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/22e25865-4487-e2aa-7556-0db30ee3c8ff/test.cache.data in system properties and HBase conf 2023-04-12 02:53:45,558 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.tmp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/22e25865-4487-e2aa-7556-0db30ee3c8ff/hadoop.tmp.dir in system properties and HBase conf 2023-04-12 02:53:45,559 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.log.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/22e25865-4487-e2aa-7556-0db30ee3c8ff/hadoop.log.dir in system properties and HBase conf 2023-04-12 02:53:45,559 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.local.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/22e25865-4487-e2aa-7556-0db30ee3c8ff/mapreduce.cluster.local.dir in system properties and HBase conf 2023-04-12 02:53:45,560 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.temp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/22e25865-4487-e2aa-7556-0db30ee3c8ff/mapreduce.cluster.temp.dir in system properties and HBase conf 2023-04-12 02:53:45,560 INFO [Time-limited test] hbase.HBaseTestingUtility(759): read short circuit is OFF 2023-04-12 02:53:45,678 WARN [Time-limited test] util.NativeCodeLoader(62): Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2023-04-12 02:53:46,108 DEBUG [Time-limited test] fs.HFileSystem(308): The file system is not a DistributedFileSystem. Skipping on block location reordering 2023-04-12 02:53:46,112 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-labels.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/22e25865-4487-e2aa-7556-0db30ee3c8ff/yarn.node-labels.fs-store.root-dir in system properties and HBase conf 2023-04-12 02:53:46,113 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-attribute.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/22e25865-4487-e2aa-7556-0db30ee3c8ff/yarn.node-attribute.fs-store.root-dir in system properties and HBase conf 2023-04-12 02:53:46,113 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.log-dirs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/22e25865-4487-e2aa-7556-0db30ee3c8ff/yarn.nodemanager.log-dirs in system properties and HBase conf 2023-04-12 02:53:46,114 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/22e25865-4487-e2aa-7556-0db30ee3c8ff/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-04-12 02:53:46,114 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.active-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/22e25865-4487-e2aa-7556-0db30ee3c8ff/yarn.timeline-service.entity-group-fs-store.active-dir in system properties and HBase conf 2023-04-12 02:53:46,115 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.done-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/22e25865-4487-e2aa-7556-0db30ee3c8ff/yarn.timeline-service.entity-group-fs-store.done-dir in system properties and HBase conf 2023-04-12 02:53:46,115 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/22e25865-4487-e2aa-7556-0db30ee3c8ff/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-04-12 02:53:46,115 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/22e25865-4487-e2aa-7556-0db30ee3c8ff/dfs.journalnode.edits.dir in system properties and HBase conf 2023-04-12 02:53:46,116 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.datanode.shared.file.descriptor.paths to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/22e25865-4487-e2aa-7556-0db30ee3c8ff/dfs.datanode.shared.file.descriptor.paths in system properties and HBase conf 2023-04-12 02:53:46,116 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting nfs.dump.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/22e25865-4487-e2aa-7556-0db30ee3c8ff/nfs.dump.dir in system properties and HBase conf 2023-04-12 02:53:46,117 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting java.io.tmpdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/22e25865-4487-e2aa-7556-0db30ee3c8ff/java.io.tmpdir in system properties and HBase conf 2023-04-12 02:53:46,117 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/22e25865-4487-e2aa-7556-0db30ee3c8ff/dfs.journalnode.edits.dir in system properties and HBase conf 2023-04-12 02:53:46,118 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.provided.aliasmap.inmemory.leveldb.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/22e25865-4487-e2aa-7556-0db30ee3c8ff/dfs.provided.aliasmap.inmemory.leveldb.dir in system properties and HBase conf 2023-04-12 02:53:46,118 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting fs.s3a.committer.staging.tmp.path to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/22e25865-4487-e2aa-7556-0db30ee3c8ff/fs.s3a.committer.staging.tmp.path in system properties and HBase conf Formatting using clusterid: testClusterID 2023-04-12 02:53:46,732 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-04-12 02:53:46,736 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-04-12 02:53:47,037 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties 2023-04-12 02:53:47,233 INFO [Time-limited test] log.Slf4jLog(67): Logging to org.slf4j.impl.Reload4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2023-04-12 02:53:47,251 WARN [Time-limited test] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-04-12 02:53:47,292 INFO [Time-limited test] log.Slf4jLog(67): jetty-6.1.26 2023-04-12 02:53:47,325 INFO [Time-limited test] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/hdfs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/22e25865-4487-e2aa-7556-0db30ee3c8ff/java.io.tmpdir/Jetty_localhost_35651_hdfs____.mb4mna/webapp 2023-04-12 02:53:47,522 INFO [Time-limited test] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:35651 2023-04-12 02:53:47,533 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-04-12 02:53:47,533 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-04-12 02:53:47,994 WARN [Listener at localhost/44691] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-04-12 02:53:48,107 WARN [Listener at localhost/44691] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-04-12 02:53:48,126 WARN [Listener at localhost/44691] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-04-12 02:53:48,133 INFO [Listener at localhost/44691] log.Slf4jLog(67): jetty-6.1.26 2023-04-12 02:53:48,138 INFO [Listener at localhost/44691] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/22e25865-4487-e2aa-7556-0db30ee3c8ff/java.io.tmpdir/Jetty_localhost_38385_datanode____.m3zani/webapp 2023-04-12 02:53:48,244 INFO [Listener at localhost/44691] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:38385 2023-04-12 02:53:48,668 WARN [Listener at localhost/39593] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-04-12 02:53:48,689 WARN [Listener at localhost/39593] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-04-12 02:53:48,699 WARN [Listener at localhost/39593] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-04-12 02:53:48,704 INFO [Listener at localhost/39593] log.Slf4jLog(67): jetty-6.1.26 2023-04-12 02:53:48,718 INFO [Listener at localhost/39593] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/22e25865-4487-e2aa-7556-0db30ee3c8ff/java.io.tmpdir/Jetty_localhost_35919_datanode____.l5uy3e/webapp 2023-04-12 02:53:48,853 INFO [Listener at localhost/39593] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:35919 2023-04-12 02:53:48,871 WARN [Listener at localhost/45897] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-04-12 02:53:48,918 WARN [Listener at localhost/45897] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-04-12 02:53:48,924 WARN [Listener at localhost/45897] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-04-12 02:53:48,926 INFO [Listener at localhost/45897] log.Slf4jLog(67): jetty-6.1.26 2023-04-12 02:53:48,934 INFO [Listener at localhost/45897] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/22e25865-4487-e2aa-7556-0db30ee3c8ff/java.io.tmpdir/Jetty_localhost_38615_datanode____.uuk2mc/webapp 2023-04-12 02:53:49,100 INFO [Listener at localhost/45897] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:38615 2023-04-12 02:53:49,124 WARN [Listener at localhost/46667] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-04-12 02:53:49,323 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xa3113da3c560b30b: Processing first storage report for DS-420b97ca-a460-40e3-b9fb-b2ada02a0355 from datanode 79922b64-644d-4757-a58c-9707ee400c82 2023-04-12 02:53:49,325 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xa3113da3c560b30b: from storage DS-420b97ca-a460-40e3-b9fb-b2ada02a0355 node DatanodeRegistration(127.0.0.1:37795, datanodeUuid=79922b64-644d-4757-a58c-9707ee400c82, infoPort=40997, infoSecurePort=0, ipcPort=45897, storageInfo=lv=-57;cid=testClusterID;nsid=2112132296;c=1681268026808), blocks: 0, hasStaleStorage: true, processing time: 2 msecs, invalidatedBlocks: 0 2023-04-12 02:53:49,325 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xc758c7f56c8ae382: Processing first storage report for DS-cdcda459-f3ea-4856-a051-b40e48414f68 from datanode 569a15e2-91b7-43b5-bb2b-d70e35911578 2023-04-12 02:53:49,325 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xc758c7f56c8ae382: from storage DS-cdcda459-f3ea-4856-a051-b40e48414f68 node DatanodeRegistration(127.0.0.1:33739, datanodeUuid=569a15e2-91b7-43b5-bb2b-d70e35911578, infoPort=34235, infoSecurePort=0, ipcPort=46667, storageInfo=lv=-57;cid=testClusterID;nsid=2112132296;c=1681268026808), blocks: 0, hasStaleStorage: true, processing time: 0 msecs, invalidatedBlocks: 0 2023-04-12 02:53:49,326 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xbb6317b8f1bb9a25: Processing first storage report for DS-737d1c12-0cb5-4776-9e46-0ca9b86b7577 from datanode ad316322-0da1-41fd-a03f-e3dd2b710c84 2023-04-12 02:53:49,326 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xbb6317b8f1bb9a25: from storage DS-737d1c12-0cb5-4776-9e46-0ca9b86b7577 node DatanodeRegistration(127.0.0.1:42215, datanodeUuid=ad316322-0da1-41fd-a03f-e3dd2b710c84, infoPort=37701, infoSecurePort=0, ipcPort=39593, storageInfo=lv=-57;cid=testClusterID;nsid=2112132296;c=1681268026808), blocks: 0, hasStaleStorage: true, processing time: 1 msecs, invalidatedBlocks: 0 2023-04-12 02:53:49,326 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xa3113da3c560b30b: Processing first storage report for DS-67dc69f6-3635-4bfd-8a64-1f440502c0f2 from datanode 79922b64-644d-4757-a58c-9707ee400c82 2023-04-12 02:53:49,326 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xa3113da3c560b30b: from storage DS-67dc69f6-3635-4bfd-8a64-1f440502c0f2 node DatanodeRegistration(127.0.0.1:37795, datanodeUuid=79922b64-644d-4757-a58c-9707ee400c82, infoPort=40997, infoSecurePort=0, ipcPort=45897, storageInfo=lv=-57;cid=testClusterID;nsid=2112132296;c=1681268026808), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-04-12 02:53:49,326 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xc758c7f56c8ae382: Processing first storage report for DS-f75d4856-6dc0-4c03-b740-384fa021a222 from datanode 569a15e2-91b7-43b5-bb2b-d70e35911578 2023-04-12 02:53:49,326 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xc758c7f56c8ae382: from storage DS-f75d4856-6dc0-4c03-b740-384fa021a222 node DatanodeRegistration(127.0.0.1:33739, datanodeUuid=569a15e2-91b7-43b5-bb2b-d70e35911578, infoPort=34235, infoSecurePort=0, ipcPort=46667, storageInfo=lv=-57;cid=testClusterID;nsid=2112132296;c=1681268026808), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-04-12 02:53:49,327 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xbb6317b8f1bb9a25: Processing first storage report for DS-5679cc77-7adb-41b8-971f-39e783af1248 from datanode ad316322-0da1-41fd-a03f-e3dd2b710c84 2023-04-12 02:53:49,327 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xbb6317b8f1bb9a25: from storage DS-5679cc77-7adb-41b8-971f-39e783af1248 node DatanodeRegistration(127.0.0.1:42215, datanodeUuid=ad316322-0da1-41fd-a03f-e3dd2b710c84, infoPort=37701, infoSecurePort=0, ipcPort=39593, storageInfo=lv=-57;cid=testClusterID;nsid=2112132296;c=1681268026808), blocks: 0, hasStaleStorage: false, processing time: 1 msecs, invalidatedBlocks: 0 2023-04-12 02:53:49,598 DEBUG [Listener at localhost/46667] hbase.HBaseTestingUtility(649): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/22e25865-4487-e2aa-7556-0db30ee3c8ff 2023-04-12 02:53:49,660 INFO [Listener at localhost/46667] zookeeper.MiniZooKeeperCluster(258): Started connectionTimeout=30000, dir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/22e25865-4487-e2aa-7556-0db30ee3c8ff/cluster_2d9c59b9-d025-040c-6051-62d17e49a4b2/zookeeper_0, clientPort=57543, secureClientPort=-1, dataDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/22e25865-4487-e2aa-7556-0db30ee3c8ff/cluster_2d9c59b9-d025-040c-6051-62d17e49a4b2/zookeeper_0/version-2, dataDirSize=424 dataLogDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/22e25865-4487-e2aa-7556-0db30ee3c8ff/cluster_2d9c59b9-d025-040c-6051-62d17e49a4b2/zookeeper_0/version-2, dataLogSize=424 tickTime=2000, maxClientCnxns=300, minSessionTimeout=4000, maxSessionTimeout=40000, serverId=0 2023-04-12 02:53:49,673 INFO [Listener at localhost/46667] zookeeper.MiniZooKeeperCluster(283): Started MiniZooKeeperCluster and ran 'stat' on client port=57543 2023-04-12 02:53:49,681 INFO [Listener at localhost/46667] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-12 02:53:49,683 INFO [Listener at localhost/46667] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-12 02:53:50,343 INFO [Listener at localhost/46667] util.FSUtils(471): Created version file at hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f with version=8 2023-04-12 02:53:50,343 INFO [Listener at localhost/46667] hbase.HBaseTestingUtility(1406): Setting hbase.fs.tmp.dir to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/hbase-staging 2023-04-12 02:53:50,672 INFO [Listener at localhost/46667] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-04-12 02:53:51,166 INFO [Listener at localhost/46667] client.ConnectionUtils(127): master/jenkins-hbase4:0 server-side Connection retries=45 2023-04-12 02:53:51,199 INFO [Listener at localhost/46667] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-12 02:53:51,199 INFO [Listener at localhost/46667] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-04-12 02:53:51,200 INFO [Listener at localhost/46667] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-04-12 02:53:51,200 INFO [Listener at localhost/46667] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-12 02:53:51,200 INFO [Listener at localhost/46667] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-04-12 02:53:51,346 INFO [Listener at localhost/46667] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-04-12 02:53:51,419 DEBUG [Listener at localhost/46667] util.ClassSize(228): Using Unsafe to estimate memory layout 2023-04-12 02:53:51,516 INFO [Listener at localhost/46667] ipc.NettyRpcServer(120): Bind to /172.31.14.131:40281 2023-04-12 02:53:51,527 INFO [Listener at localhost/46667] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-12 02:53:51,529 INFO [Listener at localhost/46667] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-12 02:53:51,550 INFO [Listener at localhost/46667] zookeeper.RecoverableZooKeeper(93): Process identifier=master:40281 connecting to ZooKeeper ensemble=127.0.0.1:57543 2023-04-12 02:53:51,592 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:402810x0, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-04-12 02:53:51,596 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): master:40281-0x101764a22930000 connected 2023-04-12 02:53:51,624 DEBUG [Listener at localhost/46667] zookeeper.ZKUtil(164): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-04-12 02:53:51,625 DEBUG [Listener at localhost/46667] zookeeper.ZKUtil(164): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-12 02:53:51,629 DEBUG [Listener at localhost/46667] zookeeper.ZKUtil(164): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-04-12 02:53:51,640 DEBUG [Listener at localhost/46667] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=40281 2023-04-12 02:53:51,641 DEBUG [Listener at localhost/46667] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=40281 2023-04-12 02:53:51,641 DEBUG [Listener at localhost/46667] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=40281 2023-04-12 02:53:51,642 DEBUG [Listener at localhost/46667] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=40281 2023-04-12 02:53:51,642 DEBUG [Listener at localhost/46667] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=40281 2023-04-12 02:53:51,649 INFO [Listener at localhost/46667] master.HMaster(444): hbase.rootdir=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f, hbase.cluster.distributed=false 2023-04-12 02:53:51,720 INFO [Listener at localhost/46667] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=45 2023-04-12 02:53:51,721 INFO [Listener at localhost/46667] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-12 02:53:51,721 INFO [Listener at localhost/46667] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-04-12 02:53:51,721 INFO [Listener at localhost/46667] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-04-12 02:53:51,721 INFO [Listener at localhost/46667] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-12 02:53:51,721 INFO [Listener at localhost/46667] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-04-12 02:53:51,726 INFO [Listener at localhost/46667] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-04-12 02:53:51,729 INFO [Listener at localhost/46667] ipc.NettyRpcServer(120): Bind to /172.31.14.131:34579 2023-04-12 02:53:51,732 INFO [Listener at localhost/46667] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-04-12 02:53:51,740 DEBUG [Listener at localhost/46667] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-04-12 02:53:51,741 INFO [Listener at localhost/46667] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-12 02:53:51,744 INFO [Listener at localhost/46667] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-12 02:53:51,746 INFO [Listener at localhost/46667] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:34579 connecting to ZooKeeper ensemble=127.0.0.1:57543 2023-04-12 02:53:51,751 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:345790x0, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-04-12 02:53:51,752 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:34579-0x101764a22930001 connected 2023-04-12 02:53:51,752 DEBUG [Listener at localhost/46667] zookeeper.ZKUtil(164): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-04-12 02:53:51,753 DEBUG [Listener at localhost/46667] zookeeper.ZKUtil(164): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-12 02:53:51,754 DEBUG [Listener at localhost/46667] zookeeper.ZKUtil(164): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-04-12 02:53:51,755 DEBUG [Listener at localhost/46667] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=34579 2023-04-12 02:53:51,755 DEBUG [Listener at localhost/46667] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=34579 2023-04-12 02:53:51,756 DEBUG [Listener at localhost/46667] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=34579 2023-04-12 02:53:51,760 DEBUG [Listener at localhost/46667] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=34579 2023-04-12 02:53:51,760 DEBUG [Listener at localhost/46667] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=34579 2023-04-12 02:53:51,773 INFO [Listener at localhost/46667] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=45 2023-04-12 02:53:51,773 INFO [Listener at localhost/46667] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-12 02:53:51,773 INFO [Listener at localhost/46667] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-04-12 02:53:51,774 INFO [Listener at localhost/46667] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-04-12 02:53:51,774 INFO [Listener at localhost/46667] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-12 02:53:51,774 INFO [Listener at localhost/46667] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-04-12 02:53:51,774 INFO [Listener at localhost/46667] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-04-12 02:53:51,775 INFO [Listener at localhost/46667] ipc.NettyRpcServer(120): Bind to /172.31.14.131:44679 2023-04-12 02:53:51,776 INFO [Listener at localhost/46667] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-04-12 02:53:51,777 DEBUG [Listener at localhost/46667] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-04-12 02:53:51,778 INFO [Listener at localhost/46667] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-12 02:53:51,781 INFO [Listener at localhost/46667] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-12 02:53:51,783 INFO [Listener at localhost/46667] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:44679 connecting to ZooKeeper ensemble=127.0.0.1:57543 2023-04-12 02:53:51,787 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:446790x0, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-04-12 02:53:51,789 DEBUG [Listener at localhost/46667] zookeeper.ZKUtil(164): regionserver:446790x0, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-04-12 02:53:51,789 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:44679-0x101764a22930002 connected 2023-04-12 02:53:51,790 DEBUG [Listener at localhost/46667] zookeeper.ZKUtil(164): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-12 02:53:51,791 DEBUG [Listener at localhost/46667] zookeeper.ZKUtil(164): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-04-12 02:53:51,792 DEBUG [Listener at localhost/46667] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=44679 2023-04-12 02:53:51,792 DEBUG [Listener at localhost/46667] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=44679 2023-04-12 02:53:51,792 DEBUG [Listener at localhost/46667] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=44679 2023-04-12 02:53:51,793 DEBUG [Listener at localhost/46667] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=44679 2023-04-12 02:53:51,793 DEBUG [Listener at localhost/46667] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=44679 2023-04-12 02:53:51,805 INFO [Listener at localhost/46667] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=45 2023-04-12 02:53:51,805 INFO [Listener at localhost/46667] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-12 02:53:51,806 INFO [Listener at localhost/46667] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-04-12 02:53:51,806 INFO [Listener at localhost/46667] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-04-12 02:53:51,806 INFO [Listener at localhost/46667] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-12 02:53:51,806 INFO [Listener at localhost/46667] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-04-12 02:53:51,806 INFO [Listener at localhost/46667] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-04-12 02:53:51,807 INFO [Listener at localhost/46667] ipc.NettyRpcServer(120): Bind to /172.31.14.131:43091 2023-04-12 02:53:51,808 INFO [Listener at localhost/46667] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-04-12 02:53:51,809 DEBUG [Listener at localhost/46667] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-04-12 02:53:51,810 INFO [Listener at localhost/46667] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-12 02:53:51,811 INFO [Listener at localhost/46667] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-12 02:53:51,812 INFO [Listener at localhost/46667] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:43091 connecting to ZooKeeper ensemble=127.0.0.1:57543 2023-04-12 02:53:51,817 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:430910x0, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-04-12 02:53:51,818 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:43091-0x101764a22930003 connected 2023-04-12 02:53:51,818 DEBUG [Listener at localhost/46667] zookeeper.ZKUtil(164): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-04-12 02:53:51,818 DEBUG [Listener at localhost/46667] zookeeper.ZKUtil(164): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-12 02:53:51,819 DEBUG [Listener at localhost/46667] zookeeper.ZKUtil(164): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-04-12 02:53:51,819 DEBUG [Listener at localhost/46667] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=43091 2023-04-12 02:53:51,820 DEBUG [Listener at localhost/46667] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=43091 2023-04-12 02:53:51,820 DEBUG [Listener at localhost/46667] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=43091 2023-04-12 02:53:51,820 DEBUG [Listener at localhost/46667] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=43091 2023-04-12 02:53:51,821 DEBUG [Listener at localhost/46667] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=43091 2023-04-12 02:53:51,822 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2168): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase4.apache.org,40281,1681268030476 2023-04-12 02:53:51,831 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-04-12 02:53:51,832 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase4.apache.org,40281,1681268030476 2023-04-12 02:53:51,851 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-04-12 02:53:51,851 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-04-12 02:53:51,851 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-12 02:53:51,851 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-04-12 02:53:51,851 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-04-12 02:53:51,853 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-04-12 02:53:51,854 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-04-12 02:53:51,854 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(227): Deleting ZNode for /hbase/backup-masters/jenkins-hbase4.apache.org,40281,1681268030476 from backup master directory 2023-04-12 02:53:51,858 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase4.apache.org,40281,1681268030476 2023-04-12 02:53:51,858 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-04-12 02:53:51,858 WARN [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-04-12 02:53:51,859 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(237): Registered as active master=jenkins-hbase4.apache.org,40281,1681268030476 2023-04-12 02:53:51,861 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating data MemStoreChunkPool with chunk size 2 MB, max count 352, initial count 0 2023-04-12 02:53:51,863 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating index MemStoreChunkPool with chunk size 204.80 KB, max count 391, initial count 0 2023-04-12 02:53:51,950 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] util.FSUtils(620): Created cluster ID file at hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/hbase.id with ID: f6fde8cd-dcec-4db9-97c7-bc0b0cd0378c 2023-04-12 02:53:51,988 INFO [master/jenkins-hbase4:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-12 02:53:52,004 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-12 02:53:52,052 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x1295f4ce to 127.0.0.1:57543 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-12 02:53:52,083 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5f9ed95f, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-12 02:53:52,106 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-12 02:53:52,107 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-04-12 02:53:52,124 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(264): ClientProtocol::create wrong number of arguments, should be hadoop 3.2 or below 2023-04-12 02:53:52,125 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(270): ClientProtocol::create wrong number of arguments, should be hadoop 2.x 2023-04-12 02:53:52,126 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(279): can not find SHOULD_REPLICATE flag, should be hadoop 2.x java.lang.IllegalArgumentException: No enum constant org.apache.hadoop.fs.CreateFlag.SHOULD_REPLICATE at java.lang.Enum.valueOf(Enum.java:238) at org.apache.hadoop.fs.CreateFlag.valueOf(CreateFlag.java:63) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.loadShouldReplicateFlag(FanOutOneBlockAsyncDFSOutputHelper.java:277) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.(FanOutOneBlockAsyncDFSOutputHelper.java:304) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:139) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-04-12 02:53:52,130 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(243): No decryptEncryptedDataEncryptionKey method in DFSClient, should be hadoop version with HDFS-12396 java.lang.NoSuchMethodException: org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(org.apache.hadoop.fs.FileEncryptionInfo) at java.lang.Class.getDeclaredMethod(Class.java:2130) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelperWithoutHDFS12396(FanOutOneBlockAsyncDFSOutputSaslHelper.java:182) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelper(FanOutOneBlockAsyncDFSOutputSaslHelper.java:241) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.(FanOutOneBlockAsyncDFSOutputSaslHelper.java:252) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:140) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-04-12 02:53:52,131 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-04-12 02:53:52,166 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7693): Creating {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, under table dir hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/MasterData/data/master/store-tmp 2023-04-12 02:53:52,211 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-12 02:53:52,211 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-04-12 02:53:52,211 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-12 02:53:52,211 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-12 02:53:52,212 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-04-12 02:53:52,212 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-12 02:53:52,212 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-12 02:53:52,212 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-04-12 02:53:52,213 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(191): WALDir=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/MasterData/WALs/jenkins-hbase4.apache.org,40281,1681268030476 2023-04-12 02:53:52,236 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C40281%2C1681268030476, suffix=, logDir=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/MasterData/WALs/jenkins-hbase4.apache.org,40281,1681268030476, archiveDir=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/MasterData/oldWALs, maxLogs=10 2023-04-12 02:53:52,305 DEBUG [RS-EventLoopGroup-5-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:33739,DS-cdcda459-f3ea-4856-a051-b40e48414f68,DISK] 2023-04-12 02:53:52,305 DEBUG [RS-EventLoopGroup-5-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:42215,DS-737d1c12-0cb5-4776-9e46-0ca9b86b7577,DISK] 2023-04-12 02:53:52,305 DEBUG [RS-EventLoopGroup-5-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:37795,DS-420b97ca-a460-40e3-b9fb-b2ada02a0355,DISK] 2023-04-12 02:53:52,314 DEBUG [RS-EventLoopGroup-5-1] asyncfs.ProtobufDecoder(123): Hadoop 3.2 and below use unshaded protobuf. java.lang.ClassNotFoundException: org.apache.hadoop.thirdparty.protobuf.MessageLite at java.net.URLClassLoader.findClass(URLClassLoader.java:387) at java.lang.ClassLoader.loadClass(ClassLoader.java:418) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) at java.lang.ClassLoader.loadClass(ClassLoader.java:351) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.io.asyncfs.ProtobufDecoder.(ProtobufDecoder.java:118) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.processWriteBlockResponse(FanOutOneBlockAsyncDFSOutputHelper.java:340) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$100(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$4.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:424) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:557) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:185) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.initialize(FanOutOneBlockAsyncDFSOutputHelper.java:418) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$300(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:476) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:471) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:625) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:105) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.fulfillConnectPromise(AbstractEpollChannel.java:653) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:691) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-12 02:53:52,401 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/MasterData/WALs/jenkins-hbase4.apache.org,40281,1681268030476/jenkins-hbase4.apache.org%2C40281%2C1681268030476.1681268032247 2023-04-12 02:53:52,402 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:37795,DS-420b97ca-a460-40e3-b9fb-b2ada02a0355,DISK], DatanodeInfoWithStorage[127.0.0.1:42215,DS-737d1c12-0cb5-4776-9e46-0ca9b86b7577,DISK], DatanodeInfoWithStorage[127.0.0.1:33739,DS-cdcda459-f3ea-4856-a051-b40e48414f68,DISK]] 2023-04-12 02:53:52,403 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7854): Opening region: {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''} 2023-04-12 02:53:52,403 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-12 02:53:52,407 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7894): checking encryption for 1595e783b53d99cd5eef43b6debb2682 2023-04-12 02:53:52,409 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7897): checking classloading for 1595e783b53d99cd5eef43b6debb2682 2023-04-12 02:53:52,475 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family proc of region 1595e783b53d99cd5eef43b6debb2682 2023-04-12 02:53:52,483 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc 2023-04-12 02:53:52,510 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc 2023-04-12 02:53:52,526 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(310): Store=1595e783b53d99cd5eef43b6debb2682/proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-12 02:53:52,535 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-04-12 02:53:52,537 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-04-12 02:53:52,554 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1055): writing seq id for 1595e783b53d99cd5eef43b6debb2682 2023-04-12 02:53:52,558 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-12 02:53:52,559 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1072): Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=65278589, jitterRate=-0.02727322280406952}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-04-12 02:53:52,560 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(965): Region open journal for 1595e783b53d99cd5eef43b6debb2682: 2023-04-12 02:53:52,561 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2023-04-12 02:53:52,582 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-04-12 02:53:52,583 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-04-12 02:53:52,585 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-04-12 02:53:52,587 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 1 msec 2023-04-12 02:53:52,620 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 33 msec 2023-04-12 02:53:52,621 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(95): Instantiated, coreThreads=3 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2023-04-12 02:53:52,646 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(253): hbase:meta replica znodes: [] 2023-04-12 02:53:52,652 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 0 possibly 'live' servers, and 0 'splitting'. 2023-04-12 02:53:52,676 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-04-12 02:53:52,681 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, ServerLocalityCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-04-12 02:53:52,682 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2023-04-12 02:53:52,689 INFO [master/jenkins-hbase4:0:becomeActiveMaster] normalizer.RegionNormalizerWorker(118): Normalizer rate limit set to unlimited 2023-04-12 02:53:52,693 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2023-04-12 02:53:52,696 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-12 02:53:52,698 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2023-04-12 02:53:52,699 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2023-04-12 02:53:52,713 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/snapshot-cleanup 2023-04-12 02:53:52,718 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-04-12 02:53:52,718 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-04-12 02:53:52,718 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-04-12 02:53:52,719 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-04-12 02:53:52,719 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-12 02:53:52,720 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(744): Active/primary master=jenkins-hbase4.apache.org,40281,1681268030476, sessionid=0x101764a22930000, setting cluster-up flag (Was=false) 2023-04-12 02:53:52,741 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-12 02:53:52,751 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2023-04-12 02:53:52,753 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,40281,1681268030476 2023-04-12 02:53:52,758 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-12 02:53:52,764 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2023-04-12 02:53:52,766 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,40281,1681268030476 2023-04-12 02:53:52,768 WARN [master/jenkins-hbase4:0:becomeActiveMaster] snapshot.SnapshotManager(302): Couldn't delete working snapshot directory: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.hbase-snapshot/.tmp 2023-04-12 02:53:52,827 INFO [RS:2;jenkins-hbase4:43091] regionserver.HRegionServer(952): ClusterId : f6fde8cd-dcec-4db9-97c7-bc0b0cd0378c 2023-04-12 02:53:52,827 INFO [RS:0;jenkins-hbase4:34579] regionserver.HRegionServer(952): ClusterId : f6fde8cd-dcec-4db9-97c7-bc0b0cd0378c 2023-04-12 02:53:52,831 INFO [RS:1;jenkins-hbase4:44679] regionserver.HRegionServer(952): ClusterId : f6fde8cd-dcec-4db9-97c7-bc0b0cd0378c 2023-04-12 02:53:52,840 DEBUG [RS:0;jenkins-hbase4:34579] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-04-12 02:53:52,840 DEBUG [RS:1;jenkins-hbase4:44679] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-04-12 02:53:52,840 DEBUG [RS:2;jenkins-hbase4:43091] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-04-12 02:53:52,852 DEBUG [RS:1;jenkins-hbase4:44679] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-04-12 02:53:52,852 DEBUG [RS:2;jenkins-hbase4:43091] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-04-12 02:53:52,852 DEBUG [RS:1;jenkins-hbase4:44679] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-04-12 02:53:52,852 DEBUG [RS:2;jenkins-hbase4:43091] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-04-12 02:53:52,857 DEBUG [RS:1;jenkins-hbase4:44679] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-04-12 02:53:52,860 DEBUG [RS:1;jenkins-hbase4:44679] zookeeper.ReadOnlyZKClient(139): Connect 0x73adb104 to 127.0.0.1:57543 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-12 02:53:52,862 DEBUG [RS:0;jenkins-hbase4:34579] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-04-12 02:53:52,862 DEBUG [RS:0;jenkins-hbase4:34579] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-04-12 02:53:52,862 DEBUG [RS:2;jenkins-hbase4:43091] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-04-12 02:53:52,866 DEBUG [RS:0;jenkins-hbase4:34579] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-04-12 02:53:52,877 DEBUG [RS:2;jenkins-hbase4:43091] zookeeper.ReadOnlyZKClient(139): Connect 0x2c37bd45 to 127.0.0.1:57543 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-12 02:53:52,877 DEBUG [RS:0;jenkins-hbase4:34579] zookeeper.ReadOnlyZKClient(139): Connect 0x1d7fbc23 to 127.0.0.1:57543 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-12 02:53:52,925 DEBUG [RS:1;jenkins-hbase4:44679] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7d6a5101, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-12 02:53:52,926 DEBUG [RS:1;jenkins-hbase4:44679] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7955da74, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-04-12 02:53:52,927 DEBUG [RS:0;jenkins-hbase4:34579] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@13b841fc, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-12 02:53:52,927 DEBUG [RS:0;jenkins-hbase4:34579] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1787d753, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-04-12 02:53:52,930 DEBUG [RS:2;jenkins-hbase4:43091] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@22f056b3, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-12 02:53:52,930 DEBUG [RS:2;jenkins-hbase4:43091] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@26cfd635, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-04-12 02:53:52,952 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT; InitMetaProcedure table=hbase:meta 2023-04-12 02:53:52,954 DEBUG [RS:2;jenkins-hbase4:43091] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:2;jenkins-hbase4:43091 2023-04-12 02:53:52,954 DEBUG [RS:0;jenkins-hbase4:34579] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:0;jenkins-hbase4:34579 2023-04-12 02:53:52,955 DEBUG [RS:1;jenkins-hbase4:44679] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:1;jenkins-hbase4:44679 2023-04-12 02:53:52,960 INFO [RS:1;jenkins-hbase4:44679] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-04-12 02:53:52,960 INFO [RS:0;jenkins-hbase4:34579] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-04-12 02:53:52,961 INFO [RS:0;jenkins-hbase4:34579] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-04-12 02:53:52,960 INFO [RS:2;jenkins-hbase4:43091] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-04-12 02:53:52,961 DEBUG [RS:0;jenkins-hbase4:34579] regionserver.HRegionServer(1023): About to register with Master. 2023-04-12 02:53:52,961 INFO [RS:1;jenkins-hbase4:44679] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-04-12 02:53:52,961 INFO [RS:2;jenkins-hbase4:43091] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-04-12 02:53:52,962 DEBUG [RS:1;jenkins-hbase4:44679] regionserver.HRegionServer(1023): About to register with Master. 2023-04-12 02:53:52,962 DEBUG [RS:2;jenkins-hbase4:43091] regionserver.HRegionServer(1023): About to register with Master. 2023-04-12 02:53:52,965 INFO [RS:2;jenkins-hbase4:43091] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,40281,1681268030476 with isa=jenkins-hbase4.apache.org/172.31.14.131:43091, startcode=1681268031804 2023-04-12 02:53:52,965 INFO [RS:1;jenkins-hbase4:44679] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,40281,1681268030476 with isa=jenkins-hbase4.apache.org/172.31.14.131:44679, startcode=1681268031772 2023-04-12 02:53:52,965 INFO [RS:0;jenkins-hbase4:34579] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,40281,1681268030476 with isa=jenkins-hbase4.apache.org/172.31.14.131:34579, startcode=1681268031719 2023-04-12 02:53:52,966 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-04-12 02:53:52,967 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-04-12 02:53:52,967 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-04-12 02:53:52,967 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-04-12 02:53:52,967 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-master/jenkins-hbase4:0, corePoolSize=10, maxPoolSize=10 2023-04-12 02:53:52,967 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-12 02:53:52,967 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_MERGE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-04-12 02:53:52,967 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-12 02:53:52,970 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1681268062970 2023-04-12 02:53:52,973 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2023-04-12 02:53:52,984 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT, locked=true; InitMetaProcedure table=hbase:meta 2023-04-12 02:53:52,984 INFO [PEWorker-1] procedure.InitMetaProcedure(71): BOOTSTRAP: creating hbase:meta region 2023-04-12 02:53:52,988 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2023-04-12 02:53:52,990 DEBUG [RS:1;jenkins-hbase4:44679] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-04-12 02:53:52,991 INFO [PEWorker-1] util.FSTableDescriptors(128): Creating new hbase:meta table descriptor 'hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-04-12 02:53:52,990 DEBUG [RS:2;jenkins-hbase4:43091] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-04-12 02:53:52,990 DEBUG [RS:0;jenkins-hbase4:34579] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-04-12 02:53:52,998 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2023-04-12 02:53:52,999 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2023-04-12 02:53:52,999 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2023-04-12 02:53:52,999 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2023-04-12 02:53:53,000 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-04-12 02:53:53,002 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2023-04-12 02:53:53,004 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2023-04-12 02:53:53,004 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2023-04-12 02:53:53,008 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2023-04-12 02:53:53,009 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2023-04-12 02:53:53,011 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(242): Starting for large file=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1681268033011,5,FailOnTimeoutGroup] 2023-04-12 02:53:53,011 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(257): Starting for small files=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1681268033011,5,FailOnTimeoutGroup] 2023-04-12 02:53:53,011 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-04-12 02:53:53,011 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1461): Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2023-04-12 02:53:53,013 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2023-04-12 02:53:53,013 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2023-04-12 02:53:53,055 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-04-12 02:53:53,056 INFO [PEWorker-1] util.FSTableDescriptors(135): Updated hbase:meta table descriptor to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-04-12 02:53:53,056 INFO [PEWorker-1] regionserver.HRegion(7675): creating {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f 2023-04-12 02:53:53,068 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:51791, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.0 (auth:SIMPLE), service=RegionServerStatusService 2023-04-12 02:53:53,068 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46407, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.1 (auth:SIMPLE), service=RegionServerStatusService 2023-04-12 02:53:53,068 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:47133, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.2 (auth:SIMPLE), service=RegionServerStatusService 2023-04-12 02:53:53,087 DEBUG [PEWorker-1] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-12 02:53:53,090 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-04-12 02:53:53,091 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=40281] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:53:53,092 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40281] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:53:53,093 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:53:53,094 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/hbase/meta/1588230740/info 2023-04-12 02:53:53,094 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-04-12 02:53:53,095 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-12 02:53:53,095 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-04-12 02:53:53,099 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/hbase/meta/1588230740/rep_barrier 2023-04-12 02:53:53,099 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-04-12 02:53:53,100 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-12 02:53:53,100 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-04-12 02:53:53,103 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/hbase/meta/1588230740/table 2023-04-12 02:53:53,103 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-04-12 02:53:53,105 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-12 02:53:53,107 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/hbase/meta/1588230740 2023-04-12 02:53:53,108 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/hbase/meta/1588230740 2023-04-12 02:53:53,111 DEBUG [RS:0;jenkins-hbase4:34579] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f 2023-04-12 02:53:53,111 DEBUG [RS:1;jenkins-hbase4:44679] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f 2023-04-12 02:53:53,111 DEBUG [RS:0;jenkins-hbase4:34579] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost:44691 2023-04-12 02:53:53,111 DEBUG [RS:2;jenkins-hbase4:43091] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f 2023-04-12 02:53:53,112 DEBUG [RS:0;jenkins-hbase4:34579] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-04-12 02:53:53,112 DEBUG [RS:1;jenkins-hbase4:44679] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost:44691 2023-04-12 02:53:53,112 DEBUG [RS:1;jenkins-hbase4:44679] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-04-12 02:53:53,112 DEBUG [RS:2;jenkins-hbase4:43091] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost:44691 2023-04-12 02:53:53,112 DEBUG [RS:2;jenkins-hbase4:43091] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-04-12 02:53:53,114 DEBUG [PEWorker-1] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-04-12 02:53:53,117 DEBUG [PEWorker-1] regionserver.HRegion(1055): writing seq id for 1588230740 2023-04-12 02:53:53,120 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-04-12 02:53:53,122 DEBUG [RS:1;jenkins-hbase4:44679] zookeeper.ZKUtil(162): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:53:53,122 DEBUG [RS:2;jenkins-hbase4:43091] zookeeper.ZKUtil(162): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:53:53,122 WARN [RS:1;jenkins-hbase4:44679] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-04-12 02:53:53,123 DEBUG [RS:0;jenkins-hbase4:34579] zookeeper.ZKUtil(162): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:53:53,123 INFO [RS:1;jenkins-hbase4:44679] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-04-12 02:53:53,123 DEBUG [RS:1;jenkins-hbase4:44679] regionserver.HRegionServer(1947): logDir=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/WALs/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:53:53,123 WARN [RS:2;jenkins-hbase4:43091] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-04-12 02:53:53,123 WARN [RS:0;jenkins-hbase4:34579] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-04-12 02:53:53,124 INFO [RS:2;jenkins-hbase4:43091] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-04-12 02:53:53,130 DEBUG [RS:2;jenkins-hbase4:43091] regionserver.HRegionServer(1947): logDir=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/WALs/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:53:53,125 INFO [RS:0;jenkins-hbase4:34579] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-04-12 02:53:53,130 DEBUG [RS:0;jenkins-hbase4:34579] regionserver.HRegionServer(1947): logDir=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/WALs/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:53:53,139 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,34579,1681268031719] 2023-04-12 02:53:53,139 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,44679,1681268031772] 2023-04-12 02:53:53,139 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,43091,1681268031804] 2023-04-12 02:53:53,144 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/hbase/meta/1588230740/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-12 02:53:53,145 DEBUG [RS:1;jenkins-hbase4:44679] zookeeper.ZKUtil(162): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:53:53,145 DEBUG [RS:0;jenkins-hbase4:34579] zookeeper.ZKUtil(162): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:53:53,146 DEBUG [RS:2;jenkins-hbase4:43091] zookeeper.ZKUtil(162): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:53:53,146 INFO [PEWorker-1] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=61694811, jitterRate=-0.08067567646503448}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-04-12 02:53:53,146 DEBUG [PEWorker-1] regionserver.HRegion(965): Region open journal for 1588230740: 2023-04-12 02:53:53,146 DEBUG [RS:1;jenkins-hbase4:44679] zookeeper.ZKUtil(162): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:53:53,146 DEBUG [RS:2;jenkins-hbase4:43091] zookeeper.ZKUtil(162): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:53:53,146 DEBUG [PEWorker-1] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-04-12 02:53:53,146 DEBUG [RS:0;jenkins-hbase4:34579] zookeeper.ZKUtil(162): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:53:53,147 INFO [PEWorker-1] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-04-12 02:53:53,147 DEBUG [PEWorker-1] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-04-12 02:53:53,147 DEBUG [RS:2;jenkins-hbase4:43091] zookeeper.ZKUtil(162): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:53:53,147 DEBUG [PEWorker-1] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-04-12 02:53:53,147 DEBUG [RS:1;jenkins-hbase4:44679] zookeeper.ZKUtil(162): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:53:53,147 DEBUG [RS:0;jenkins-hbase4:34579] zookeeper.ZKUtil(162): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:53:53,147 DEBUG [PEWorker-1] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-04-12 02:53:53,149 INFO [PEWorker-1] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-04-12 02:53:53,149 DEBUG [PEWorker-1] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-04-12 02:53:53,155 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_ASSIGN_META, locked=true; InitMetaProcedure table=hbase:meta 2023-04-12 02:53:53,155 INFO [PEWorker-1] procedure.InitMetaProcedure(103): Going to assign meta 2023-04-12 02:53:53,159 DEBUG [RS:1;jenkins-hbase4:44679] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-04-12 02:53:53,159 DEBUG [RS:2;jenkins-hbase4:43091] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-04-12 02:53:53,159 DEBUG [RS:0;jenkins-hbase4:34579] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-04-12 02:53:53,166 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-04-12 02:53:53,170 INFO [RS:2;jenkins-hbase4:43091] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-04-12 02:53:53,171 INFO [RS:1;jenkins-hbase4:44679] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-04-12 02:53:53,170 INFO [RS:0;jenkins-hbase4:34579] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-04-12 02:53:53,182 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-04-12 02:53:53,185 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OFFLINE, location=null; forceNewPlan=false, retain=false 2023-04-12 02:53:53,194 INFO [RS:1;jenkins-hbase4:44679] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-04-12 02:53:53,196 INFO [RS:2;jenkins-hbase4:43091] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-04-12 02:53:53,196 INFO [RS:0;jenkins-hbase4:34579] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-04-12 02:53:53,198 INFO [RS:0;jenkins-hbase4:34579] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-04-12 02:53:53,198 INFO [RS:1;jenkins-hbase4:44679] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-04-12 02:53:53,199 INFO [RS:0;jenkins-hbase4:34579] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-04-12 02:53:53,198 INFO [RS:2;jenkins-hbase4:43091] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-04-12 02:53:53,199 INFO [RS:1;jenkins-hbase4:44679] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-04-12 02:53:53,199 INFO [RS:2;jenkins-hbase4:43091] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-04-12 02:53:53,200 INFO [RS:0;jenkins-hbase4:34579] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-04-12 02:53:53,200 INFO [RS:1;jenkins-hbase4:44679] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-04-12 02:53:53,200 INFO [RS:2;jenkins-hbase4:43091] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-04-12 02:53:53,208 INFO [RS:2;jenkins-hbase4:43091] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-04-12 02:53:53,208 INFO [RS:0;jenkins-hbase4:34579] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-04-12 02:53:53,208 INFO [RS:1;jenkins-hbase4:44679] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-04-12 02:53:53,208 DEBUG [RS:2;jenkins-hbase4:43091] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-12 02:53:53,208 DEBUG [RS:0;jenkins-hbase4:34579] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-12 02:53:53,209 DEBUG [RS:2;jenkins-hbase4:43091] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-12 02:53:53,209 DEBUG [RS:0;jenkins-hbase4:34579] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-12 02:53:53,209 DEBUG [RS:2;jenkins-hbase4:43091] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-12 02:53:53,209 DEBUG [RS:0;jenkins-hbase4:34579] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-12 02:53:53,209 DEBUG [RS:2;jenkins-hbase4:43091] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-12 02:53:53,209 DEBUG [RS:0;jenkins-hbase4:34579] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-12 02:53:53,209 DEBUG [RS:2;jenkins-hbase4:43091] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-12 02:53:53,209 DEBUG [RS:0;jenkins-hbase4:34579] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-12 02:53:53,209 DEBUG [RS:2;jenkins-hbase4:43091] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-04-12 02:53:53,209 DEBUG [RS:1;jenkins-hbase4:44679] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-12 02:53:53,209 DEBUG [RS:2;jenkins-hbase4:43091] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-12 02:53:53,209 DEBUG [RS:0;jenkins-hbase4:34579] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-04-12 02:53:53,209 DEBUG [RS:2;jenkins-hbase4:43091] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-12 02:53:53,209 DEBUG [RS:1;jenkins-hbase4:44679] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-12 02:53:53,210 DEBUG [RS:2;jenkins-hbase4:43091] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-12 02:53:53,210 DEBUG [RS:1;jenkins-hbase4:44679] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-12 02:53:53,210 DEBUG [RS:2;jenkins-hbase4:43091] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-12 02:53:53,210 DEBUG [RS:1;jenkins-hbase4:44679] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-12 02:53:53,209 DEBUG [RS:0;jenkins-hbase4:34579] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-12 02:53:53,210 DEBUG [RS:1;jenkins-hbase4:44679] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-12 02:53:53,210 DEBUG [RS:0;jenkins-hbase4:34579] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-12 02:53:53,210 DEBUG [RS:1;jenkins-hbase4:44679] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-04-12 02:53:53,210 DEBUG [RS:0;jenkins-hbase4:34579] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-12 02:53:53,210 DEBUG [RS:1;jenkins-hbase4:44679] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-12 02:53:53,210 DEBUG [RS:0;jenkins-hbase4:34579] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-12 02:53:53,210 DEBUG [RS:1;jenkins-hbase4:44679] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-12 02:53:53,210 DEBUG [RS:1;jenkins-hbase4:44679] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-12 02:53:53,211 DEBUG [RS:1;jenkins-hbase4:44679] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-12 02:53:53,211 INFO [RS:0;jenkins-hbase4:34579] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-04-12 02:53:53,211 INFO [RS:0;jenkins-hbase4:34579] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-04-12 02:53:53,211 INFO [RS:0;jenkins-hbase4:34579] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-04-12 02:53:53,212 INFO [RS:2;jenkins-hbase4:43091] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-04-12 02:53:53,212 INFO [RS:2;jenkins-hbase4:43091] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-04-12 02:53:53,212 INFO [RS:2;jenkins-hbase4:43091] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-04-12 02:53:53,215 INFO [RS:1;jenkins-hbase4:44679] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-04-12 02:53:53,216 INFO [RS:1;jenkins-hbase4:44679] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-04-12 02:53:53,216 INFO [RS:1;jenkins-hbase4:44679] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-04-12 02:53:53,228 INFO [RS:0;jenkins-hbase4:34579] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-04-12 02:53:53,228 INFO [RS:2;jenkins-hbase4:43091] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-04-12 02:53:53,231 INFO [RS:2;jenkins-hbase4:43091] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,43091,1681268031804-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-04-12 02:53:53,231 INFO [RS:0;jenkins-hbase4:34579] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,34579,1681268031719-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-04-12 02:53:53,231 INFO [RS:1;jenkins-hbase4:44679] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-04-12 02:53:53,232 INFO [RS:1;jenkins-hbase4:44679] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,44679,1681268031772-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-04-12 02:53:53,249 INFO [RS:2;jenkins-hbase4:43091] regionserver.Replication(203): jenkins-hbase4.apache.org,43091,1681268031804 started 2023-04-12 02:53:53,249 INFO [RS:2;jenkins-hbase4:43091] regionserver.HRegionServer(1638): Serving as jenkins-hbase4.apache.org,43091,1681268031804, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:43091, sessionid=0x101764a22930003 2023-04-12 02:53:53,249 DEBUG [RS:2;jenkins-hbase4:43091] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-04-12 02:53:53,250 DEBUG [RS:2;jenkins-hbase4:43091] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:53:53,250 DEBUG [RS:2;jenkins-hbase4:43091] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,43091,1681268031804' 2023-04-12 02:53:53,250 DEBUG [RS:2;jenkins-hbase4:43091] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:53:53,250 DEBUG [RS:2;jenkins-hbase4:43091] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:53:53,251 DEBUG [RS:2;jenkins-hbase4:43091] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-04-12 02:53:53,251 DEBUG [RS:2;jenkins-hbase4:43091] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-04-12 02:53:53,251 DEBUG [RS:2;jenkins-hbase4:43091] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:53:53,251 DEBUG [RS:2;jenkins-hbase4:43091] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,43091,1681268031804' 2023-04-12 02:53:53,251 DEBUG [RS:2;jenkins-hbase4:43091] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-04-12 02:53:53,251 INFO [RS:0;jenkins-hbase4:34579] regionserver.Replication(203): jenkins-hbase4.apache.org,34579,1681268031719 started 2023-04-12 02:53:53,251 INFO [RS:1;jenkins-hbase4:44679] regionserver.Replication(203): jenkins-hbase4.apache.org,44679,1681268031772 started 2023-04-12 02:53:53,252 INFO [RS:0;jenkins-hbase4:34579] regionserver.HRegionServer(1638): Serving as jenkins-hbase4.apache.org,34579,1681268031719, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:34579, sessionid=0x101764a22930001 2023-04-12 02:53:53,252 INFO [RS:1;jenkins-hbase4:44679] regionserver.HRegionServer(1638): Serving as jenkins-hbase4.apache.org,44679,1681268031772, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:44679, sessionid=0x101764a22930002 2023-04-12 02:53:53,252 DEBUG [RS:0;jenkins-hbase4:34579] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-04-12 02:53:53,252 DEBUG [RS:1;jenkins-hbase4:44679] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-04-12 02:53:53,252 DEBUG [RS:1;jenkins-hbase4:44679] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:53:53,252 DEBUG [RS:0;jenkins-hbase4:34579] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:53:53,252 DEBUG [RS:1;jenkins-hbase4:44679] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,44679,1681268031772' 2023-04-12 02:53:53,253 DEBUG [RS:1;jenkins-hbase4:44679] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:53:53,252 DEBUG [RS:2;jenkins-hbase4:43091] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-04-12 02:53:53,253 DEBUG [RS:0;jenkins-hbase4:34579] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,34579,1681268031719' 2023-04-12 02:53:53,253 DEBUG [RS:0;jenkins-hbase4:34579] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:53:53,253 DEBUG [RS:1;jenkins-hbase4:44679] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:53:53,253 DEBUG [RS:2;jenkins-hbase4:43091] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-04-12 02:53:53,254 INFO [RS:2;jenkins-hbase4:43091] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-04-12 02:53:53,254 DEBUG [RS:0;jenkins-hbase4:34579] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:53:53,254 INFO [RS:2;jenkins-hbase4:43091] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-04-12 02:53:53,254 DEBUG [RS:1;jenkins-hbase4:44679] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-04-12 02:53:53,254 DEBUG [RS:1;jenkins-hbase4:44679] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-04-12 02:53:53,254 DEBUG [RS:1;jenkins-hbase4:44679] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:53:53,254 DEBUG [RS:1;jenkins-hbase4:44679] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,44679,1681268031772' 2023-04-12 02:53:53,254 DEBUG [RS:1;jenkins-hbase4:44679] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-04-12 02:53:53,254 DEBUG [RS:0;jenkins-hbase4:34579] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-04-12 02:53:53,255 DEBUG [RS:0;jenkins-hbase4:34579] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-04-12 02:53:53,255 DEBUG [RS:0;jenkins-hbase4:34579] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:53:53,255 DEBUG [RS:0;jenkins-hbase4:34579] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,34579,1681268031719' 2023-04-12 02:53:53,255 DEBUG [RS:0;jenkins-hbase4:34579] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-04-12 02:53:53,255 DEBUG [RS:1;jenkins-hbase4:44679] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-04-12 02:53:53,255 DEBUG [RS:0;jenkins-hbase4:34579] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-04-12 02:53:53,255 DEBUG [RS:1;jenkins-hbase4:44679] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-04-12 02:53:53,255 INFO [RS:1;jenkins-hbase4:44679] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-04-12 02:53:53,256 INFO [RS:1;jenkins-hbase4:44679] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-04-12 02:53:53,257 DEBUG [RS:0;jenkins-hbase4:34579] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-04-12 02:53:53,257 INFO [RS:0;jenkins-hbase4:34579] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-04-12 02:53:53,257 INFO [RS:0;jenkins-hbase4:34579] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-04-12 02:53:53,337 DEBUG [jenkins-hbase4:40281] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=3, allServersCount=3 2023-04-12 02:53:53,341 DEBUG [jenkins-hbase4:40281] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-04-12 02:53:53,347 DEBUG [jenkins-hbase4:40281] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-04-12 02:53:53,347 DEBUG [jenkins-hbase4:40281] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-04-12 02:53:53,347 DEBUG [jenkins-hbase4:40281] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-04-12 02:53:53,347 DEBUG [jenkins-hbase4:40281] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-04-12 02:53:53,350 INFO [PEWorker-1] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,44679,1681268031772, state=OPENING 2023-04-12 02:53:53,357 DEBUG [PEWorker-1] zookeeper.MetaTableLocator(240): hbase:meta region location doesn't exist, create it 2023-04-12 02:53:53,358 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-12 02:53:53,359 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-04-12 02:53:53,362 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=3, ppid=2, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,44679,1681268031772}] 2023-04-12 02:53:53,365 INFO [RS:2;jenkins-hbase4:43091] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C43091%2C1681268031804, suffix=, logDir=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/WALs/jenkins-hbase4.apache.org,43091,1681268031804, archiveDir=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/oldWALs, maxLogs=32 2023-04-12 02:53:53,365 INFO [RS:1;jenkins-hbase4:44679] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C44679%2C1681268031772, suffix=, logDir=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/WALs/jenkins-hbase4.apache.org,44679,1681268031772, archiveDir=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/oldWALs, maxLogs=32 2023-04-12 02:53:53,365 INFO [RS:0;jenkins-hbase4:34579] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C34579%2C1681268031719, suffix=, logDir=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/WALs/jenkins-hbase4.apache.org,34579,1681268031719, archiveDir=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/oldWALs, maxLogs=32 2023-04-12 02:53:53,404 DEBUG [RS-EventLoopGroup-5-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:33739,DS-cdcda459-f3ea-4856-a051-b40e48414f68,DISK] 2023-04-12 02:53:53,404 DEBUG [RS-EventLoopGroup-5-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:42215,DS-737d1c12-0cb5-4776-9e46-0ca9b86b7577,DISK] 2023-04-12 02:53:53,416 DEBUG [RS-EventLoopGroup-5-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:37795,DS-420b97ca-a460-40e3-b9fb-b2ada02a0355,DISK] 2023-04-12 02:53:53,417 DEBUG [RS-EventLoopGroup-5-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:37795,DS-420b97ca-a460-40e3-b9fb-b2ada02a0355,DISK] 2023-04-12 02:53:53,417 DEBUG [RS-EventLoopGroup-5-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:33739,DS-cdcda459-f3ea-4856-a051-b40e48414f68,DISK] 2023-04-12 02:53:53,417 DEBUG [RS-EventLoopGroup-5-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:42215,DS-737d1c12-0cb5-4776-9e46-0ca9b86b7577,DISK] 2023-04-12 02:53:53,420 DEBUG [RS-EventLoopGroup-5-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:37795,DS-420b97ca-a460-40e3-b9fb-b2ada02a0355,DISK] 2023-04-12 02:53:53,420 DEBUG [RS-EventLoopGroup-5-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:42215,DS-737d1c12-0cb5-4776-9e46-0ca9b86b7577,DISK] 2023-04-12 02:53:53,420 DEBUG [RS-EventLoopGroup-5-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:33739,DS-cdcda459-f3ea-4856-a051-b40e48414f68,DISK] 2023-04-12 02:53:53,431 INFO [RS:0;jenkins-hbase4:34579] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/WALs/jenkins-hbase4.apache.org,34579,1681268031719/jenkins-hbase4.apache.org%2C34579%2C1681268031719.1681268033370 2023-04-12 02:53:53,434 DEBUG [RS:0;jenkins-hbase4:34579] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:42215,DS-737d1c12-0cb5-4776-9e46-0ca9b86b7577,DISK], DatanodeInfoWithStorage[127.0.0.1:37795,DS-420b97ca-a460-40e3-b9fb-b2ada02a0355,DISK], DatanodeInfoWithStorage[127.0.0.1:33739,DS-cdcda459-f3ea-4856-a051-b40e48414f68,DISK]] 2023-04-12 02:53:53,434 INFO [RS:1;jenkins-hbase4:44679] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/WALs/jenkins-hbase4.apache.org,44679,1681268031772/jenkins-hbase4.apache.org%2C44679%2C1681268031772.1681268033369 2023-04-12 02:53:53,437 DEBUG [RS:1;jenkins-hbase4:44679] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:42215,DS-737d1c12-0cb5-4776-9e46-0ca9b86b7577,DISK], DatanodeInfoWithStorage[127.0.0.1:37795,DS-420b97ca-a460-40e3-b9fb-b2ada02a0355,DISK], DatanodeInfoWithStorage[127.0.0.1:33739,DS-cdcda459-f3ea-4856-a051-b40e48414f68,DISK]] 2023-04-12 02:53:53,444 INFO [RS:2;jenkins-hbase4:43091] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/WALs/jenkins-hbase4.apache.org,43091,1681268031804/jenkins-hbase4.apache.org%2C43091%2C1681268031804.1681268033369 2023-04-12 02:53:53,444 DEBUG [RS:2;jenkins-hbase4:43091] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:42215,DS-737d1c12-0cb5-4776-9e46-0ca9b86b7577,DISK], DatanodeInfoWithStorage[127.0.0.1:33739,DS-cdcda459-f3ea-4856-a051-b40e48414f68,DISK], DatanodeInfoWithStorage[127.0.0.1:37795,DS-420b97ca-a460-40e3-b9fb-b2ada02a0355,DISK]] 2023-04-12 02:53:53,547 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:53:53,550 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-04-12 02:53:53,553 INFO [RS-EventLoopGroup-4-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43770, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-04-12 02:53:53,570 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open hbase:meta,,1.1588230740 2023-04-12 02:53:53,570 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-04-12 02:53:53,573 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C44679%2C1681268031772.meta, suffix=.meta, logDir=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/WALs/jenkins-hbase4.apache.org,44679,1681268031772, archiveDir=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/oldWALs, maxLogs=32 2023-04-12 02:53:53,593 DEBUG [RS-EventLoopGroup-5-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:37795,DS-420b97ca-a460-40e3-b9fb-b2ada02a0355,DISK] 2023-04-12 02:53:53,595 DEBUG [RS-EventLoopGroup-5-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:42215,DS-737d1c12-0cb5-4776-9e46-0ca9b86b7577,DISK] 2023-04-12 02:53:53,595 DEBUG [RS-EventLoopGroup-5-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:33739,DS-cdcda459-f3ea-4856-a051-b40e48414f68,DISK] 2023-04-12 02:53:53,601 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/WALs/jenkins-hbase4.apache.org,44679,1681268031772/jenkins-hbase4.apache.org%2C44679%2C1681268031772.meta.1681268033574.meta 2023-04-12 02:53:53,602 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:37795,DS-420b97ca-a460-40e3-b9fb-b2ada02a0355,DISK], DatanodeInfoWithStorage[127.0.0.1:42215,DS-737d1c12-0cb5-4776-9e46-0ca9b86b7577,DISK], DatanodeInfoWithStorage[127.0.0.1:33739,DS-cdcda459-f3ea-4856-a051-b40e48414f68,DISK]] 2023-04-12 02:53:53,602 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-04-12 02:53:53,606 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(174): System coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro loaded, priority=536870911. 2023-04-12 02:53:53,607 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-04-12 02:53:53,624 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(8550): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-04-12 02:53:53,627 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-04-12 02:53:53,632 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-04-12 02:53:53,632 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-12 02:53:53,633 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 1588230740 2023-04-12 02:53:53,633 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 1588230740 2023-04-12 02:53:53,636 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-04-12 02:53:53,638 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/hbase/meta/1588230740/info 2023-04-12 02:53:53,638 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/hbase/meta/1588230740/info 2023-04-12 02:53:53,639 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-04-12 02:53:53,640 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-12 02:53:53,640 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-04-12 02:53:53,642 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/hbase/meta/1588230740/rep_barrier 2023-04-12 02:53:53,642 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/hbase/meta/1588230740/rep_barrier 2023-04-12 02:53:53,643 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-04-12 02:53:53,644 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-12 02:53:53,644 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-04-12 02:53:53,645 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/hbase/meta/1588230740/table 2023-04-12 02:53:53,645 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/hbase/meta/1588230740/table 2023-04-12 02:53:53,646 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-04-12 02:53:53,647 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-12 02:53:53,650 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/hbase/meta/1588230740 2023-04-12 02:53:53,652 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/hbase/meta/1588230740 2023-04-12 02:53:53,656 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-04-12 02:53:53,658 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 1588230740 2023-04-12 02:53:53,659 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=63835620, jitterRate=-0.04877513647079468}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-04-12 02:53:53,659 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 1588230740: 2023-04-12 02:53:53,674 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:meta,,1.1588230740, pid=3, masterSystemTime=1681268033540 2023-04-12 02:53:53,691 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:meta,,1.1588230740 2023-04-12 02:53:53,692 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened hbase:meta,,1.1588230740 2023-04-12 02:53:53,692 INFO [PEWorker-1] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,44679,1681268031772, state=OPEN 2023-04-12 02:53:53,695 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-04-12 02:53:53,695 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-04-12 02:53:53,699 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=3, resume processing ppid=2 2023-04-12 02:53:53,699 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=3, ppid=2, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,44679,1681268031772 in 333 msec 2023-04-12 02:53:53,705 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=2, resume processing ppid=1 2023-04-12 02:53:53,705 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=2, ppid=1, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 536 msec 2023-04-12 02:53:53,710 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta in 896 msec 2023-04-12 02:53:53,710 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(953): Wait for region servers to report in: status=null, state=RUNNING, startTime=1681268033710, completionTime=-1 2023-04-12 02:53:53,711 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ServerManager(821): Finished waiting on RegionServer count=3; waited=0ms, expected min=3 server(s), max=3 server(s), master is running 2023-04-12 02:53:53,711 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1517): Joining cluster... 2023-04-12 02:53:53,780 DEBUG [hconnection-0x6cb27db9-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-12 02:53:53,783 INFO [RS-EventLoopGroup-4-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43780, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-12 02:53:53,801 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1529): Number of RegionServers=3 2023-04-12 02:53:53,801 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1681268093801 2023-04-12 02:53:53,801 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1681268153801 2023-04-12 02:53:53,801 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1536): Joined the cluster in 90 msec 2023-04-12 02:53:53,827 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,40281,1681268030476-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2023-04-12 02:53:53,828 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,40281,1681268030476-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2023-04-12 02:53:53,828 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,40281,1681268030476-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2023-04-12 02:53:53,830 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=CatalogJanitor-jenkins-hbase4:40281, period=300000, unit=MILLISECONDS is enabled. 2023-04-12 02:53:53,831 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2023-04-12 02:53:53,843 DEBUG [master/jenkins-hbase4:0.Chore.1] janitor.CatalogJanitor(175): 2023-04-12 02:53:53,851 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.TableNamespaceManager(92): Namespace table not found. Creating... 2023-04-12 02:53:53,853 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2148): Client=null/null create 'hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-04-12 02:53:53,864 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2023-04-12 02:53:53,867 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_PRE_OPERATION 2023-04-12 02:53:53,870 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-12 02:53:53,897 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/hbase/namespace/1f255e980a31bb266f2206a30aa2ae3f 2023-04-12 02:53:53,899 DEBUG [HFileArchiver-1] backup.HFileArchiver(153): Directory hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/hbase/namespace/1f255e980a31bb266f2206a30aa2ae3f empty. 2023-04-12 02:53:53,900 DEBUG [HFileArchiver-1] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/hbase/namespace/1f255e980a31bb266f2206a30aa2ae3f 2023-04-12 02:53:53,901 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived hbase:namespace regions 2023-04-12 02:53:53,941 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 2023-04-12 02:53:53,944 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(7675): creating {ENCODED => 1f255e980a31bb266f2206a30aa2ae3f, NAME => 'hbase:namespace,,1681268033852.1f255e980a31bb266f2206a30aa2ae3f.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp 2023-04-12 02:53:53,967 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1681268033852.1f255e980a31bb266f2206a30aa2ae3f.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-12 02:53:53,967 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1604): Closing 1f255e980a31bb266f2206a30aa2ae3f, disabling compactions & flushes 2023-04-12 02:53:53,967 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1681268033852.1f255e980a31bb266f2206a30aa2ae3f. 2023-04-12 02:53:53,967 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1681268033852.1f255e980a31bb266f2206a30aa2ae3f. 2023-04-12 02:53:53,967 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1681268033852.1f255e980a31bb266f2206a30aa2ae3f. after waiting 0 ms 2023-04-12 02:53:53,967 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1681268033852.1f255e980a31bb266f2206a30aa2ae3f. 2023-04-12 02:53:53,967 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1838): Closed hbase:namespace,,1681268033852.1f255e980a31bb266f2206a30aa2ae3f. 2023-04-12 02:53:53,968 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1558): Region close journal for 1f255e980a31bb266f2206a30aa2ae3f: 2023-04-12 02:53:53,973 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ADD_TO_META 2023-04-12 02:53:53,990 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"hbase:namespace,,1681268033852.1f255e980a31bb266f2206a30aa2ae3f.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1681268033975"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681268033975"}]},"ts":"1681268033975"} 2023-04-12 02:53:54,022 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-12 02:53:54,024 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-12 02:53:54,031 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681268034024"}]},"ts":"1681268034024"} 2023-04-12 02:53:54,036 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLING in hbase:meta 2023-04-12 02:53:54,041 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-04-12 02:53:54,042 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-04-12 02:53:54,042 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-04-12 02:53:54,042 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-04-12 02:53:54,042 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-04-12 02:53:54,045 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=1f255e980a31bb266f2206a30aa2ae3f, ASSIGN}] 2023-04-12 02:53:54,049 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=1f255e980a31bb266f2206a30aa2ae3f, ASSIGN 2023-04-12 02:53:54,054 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=1f255e980a31bb266f2206a30aa2ae3f, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,44679,1681268031772; forceNewPlan=false, retain=false 2023-04-12 02:53:54,206 INFO [jenkins-hbase4:40281] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-04-12 02:53:54,207 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=1f255e980a31bb266f2206a30aa2ae3f, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:53:54,207 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1681268033852.1f255e980a31bb266f2206a30aa2ae3f.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1681268034206"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681268034206"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681268034206"}]},"ts":"1681268034206"} 2023-04-12 02:53:54,211 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=6, ppid=5, state=RUNNABLE; OpenRegionProcedure 1f255e980a31bb266f2206a30aa2ae3f, server=jenkins-hbase4.apache.org,44679,1681268031772}] 2023-04-12 02:53:54,375 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open hbase:namespace,,1681268033852.1f255e980a31bb266f2206a30aa2ae3f. 2023-04-12 02:53:54,375 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1f255e980a31bb266f2206a30aa2ae3f, NAME => 'hbase:namespace,,1681268033852.1f255e980a31bb266f2206a30aa2ae3f.', STARTKEY => '', ENDKEY => ''} 2023-04-12 02:53:54,376 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(174): System coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro loaded, priority=536870911. 2023-04-12 02:53:54,377 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace 1f255e980a31bb266f2206a30aa2ae3f 2023-04-12 02:53:54,377 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1681268033852.1f255e980a31bb266f2206a30aa2ae3f.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-12 02:53:54,377 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 1f255e980a31bb266f2206a30aa2ae3f 2023-04-12 02:53:54,377 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 1f255e980a31bb266f2206a30aa2ae3f 2023-04-12 02:53:54,379 INFO [StoreOpener-1f255e980a31bb266f2206a30aa2ae3f-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1f255e980a31bb266f2206a30aa2ae3f 2023-04-12 02:53:54,381 DEBUG [StoreOpener-1f255e980a31bb266f2206a30aa2ae3f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/hbase/namespace/1f255e980a31bb266f2206a30aa2ae3f/info 2023-04-12 02:53:54,381 DEBUG [StoreOpener-1f255e980a31bb266f2206a30aa2ae3f-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/hbase/namespace/1f255e980a31bb266f2206a30aa2ae3f/info 2023-04-12 02:53:54,382 INFO [StoreOpener-1f255e980a31bb266f2206a30aa2ae3f-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1f255e980a31bb266f2206a30aa2ae3f columnFamilyName info 2023-04-12 02:53:54,383 INFO [StoreOpener-1f255e980a31bb266f2206a30aa2ae3f-1] regionserver.HStore(310): Store=1f255e980a31bb266f2206a30aa2ae3f/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-12 02:53:54,384 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/hbase/namespace/1f255e980a31bb266f2206a30aa2ae3f 2023-04-12 02:53:54,385 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/hbase/namespace/1f255e980a31bb266f2206a30aa2ae3f 2023-04-12 02:53:54,390 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 1f255e980a31bb266f2206a30aa2ae3f 2023-04-12 02:53:54,394 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/hbase/namespace/1f255e980a31bb266f2206a30aa2ae3f/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-12 02:53:54,394 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 1f255e980a31bb266f2206a30aa2ae3f; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=59126340, jitterRate=-0.11894887685775757}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-04-12 02:53:54,394 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 1f255e980a31bb266f2206a30aa2ae3f: 2023-04-12 02:53:54,397 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:namespace,,1681268033852.1f255e980a31bb266f2206a30aa2ae3f., pid=6, masterSystemTime=1681268034365 2023-04-12 02:53:54,400 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:namespace,,1681268033852.1f255e980a31bb266f2206a30aa2ae3f. 2023-04-12 02:53:54,400 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened hbase:namespace,,1681268033852.1f255e980a31bb266f2206a30aa2ae3f. 2023-04-12 02:53:54,402 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=1f255e980a31bb266f2206a30aa2ae3f, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:53:54,402 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1681268033852.1f255e980a31bb266f2206a30aa2ae3f.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1681268034401"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1681268034401"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681268034401"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681268034401"}]},"ts":"1681268034401"} 2023-04-12 02:53:54,409 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=6, resume processing ppid=5 2023-04-12 02:53:54,409 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=6, ppid=5, state=SUCCESS; OpenRegionProcedure 1f255e980a31bb266f2206a30aa2ae3f, server=jenkins-hbase4.apache.org,44679,1681268031772 in 194 msec 2023-04-12 02:53:54,413 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=5, resume processing ppid=4 2023-04-12 02:53:54,413 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=5, ppid=4, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=1f255e980a31bb266f2206a30aa2ae3f, ASSIGN in 366 msec 2023-04-12 02:53:54,415 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-12 02:53:54,415 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681268034415"}]},"ts":"1681268034415"} 2023-04-12 02:53:54,417 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLED in hbase:meta 2023-04-12 02:53:54,423 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_POST_OPERATION 2023-04-12 02:53:54,425 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace in 569 msec 2023-04-12 02:53:54,469 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/namespace 2023-04-12 02:53:54,471 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/namespace 2023-04-12 02:53:54,471 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-12 02:53:54,539 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=7, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2023-04-12 02:53:54,552 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-04-12 02:53:54,559 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default in 28 msec 2023-04-12 02:53:54,573 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=8, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2023-04-12 02:53:54,586 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-04-12 02:53:54,591 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase in 18 msec 2023-04-12 02:53:54,598 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-04-12 02:53:54,601 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2023-04-12 02:53:54,602 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1083): Master has completed initialization 2.743sec 2023-04-12 02:53:54,604 INFO [master/jenkins-hbase4:0:becomeActiveMaster] quotas.MasterQuotaManager(97): Quota support disabled 2023-04-12 02:53:54,606 INFO [master/jenkins-hbase4:0:becomeActiveMaster] slowlog.SlowLogMasterService(57): Slow/Large requests logging to system table hbase:slowlog is disabled. Quitting. 2023-04-12 02:53:54,606 INFO [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKWatcher(269): not a secure deployment, proceeding 2023-04-12 02:53:54,607 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,40281,1681268030476-ExpiredMobFileCleanerChore, period=86400, unit=SECONDS is enabled. 2023-04-12 02:53:54,608 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,40281,1681268030476-MobCompactionChore, period=604800, unit=SECONDS is enabled. 2023-04-12 02:53:54,616 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1175): Balancer post startup initialization complete, took 0 seconds 2023-04-12 02:53:54,637 DEBUG [Listener at localhost/46667] zookeeper.ReadOnlyZKClient(139): Connect 0x7b7d11f1 to 127.0.0.1:57543 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-12 02:53:54,643 DEBUG [Listener at localhost/46667] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@fdafbe7, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-12 02:53:54,659 DEBUG [hconnection-0x2d8012cd-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-12 02:53:54,671 INFO [RS-EventLoopGroup-4-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43794, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-12 02:53:54,681 INFO [Listener at localhost/46667] hbase.HBaseTestingUtility(1145): Minicluster is up; activeMaster=jenkins-hbase4.apache.org,40281,1681268030476 2023-04-12 02:53:54,711 INFO [Listener at localhost/46667] hbase.ResourceChecker(147): before: master.assignment.TestSplitTableRegionProcedure#testSplitTableRegionUnevenDaughter Thread=359, OpenFileDescriptor=691, MaxFileDescriptor=60000, SystemLoadAverage=340, ProcessCount=169, AvailableMemoryMB=1739 2023-04-12 02:53:54,716 WARN [Listener at localhost/46667] procedure2.ProcedureTestingUtility(193): Set Kill before store update to: false 2023-04-12 02:53:54,724 DEBUG [Listener at localhost/46667] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-12 02:53:54,727 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:44732, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-12 02:53:54,741 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/balancer 2023-04-12 02:53:54,741 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-12 02:53:54,742 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40281] master.MasterRpcServices(492): Client=jenkins//172.31.14.131 set balanceSwitch=false 2023-04-12 02:53:54,747 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(1029): Stored pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-12 02:53:54,749 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=testSplitTableRegionUnevenDaughter execute state=CREATE_TABLE_PRE_OPERATION 2023-04-12 02:53:54,751 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=testSplitTableRegionUnevenDaughter execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-12 02:53:54,754 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4 2023-04-12 02:53:54,755 DEBUG [HFileArchiver-2] backup.HFileArchiver(153): Directory hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4 empty. 2023-04-12 02:53:54,757 DEBUG [HFileArchiver-2] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4 2023-04-12 02:53:54,757 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived testSplitTableRegionUnevenDaughter regions 2023-04-12 02:53:54,777 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitTableRegionUnevenDaughter/.tabledesc/.tableinfo.0000000001 2023-04-12 02:53:54,779 INFO [RegionOpenAndInit-testSplitTableRegionUnevenDaughter-pool-0] regionserver.HRegion(7675): creating {ENCODED => cc9012ec0b656baa24d04368fbe026c4, NAME => 'testSplitTableRegionUnevenDaughter,,1681268034745.cc9012ec0b656baa24d04368fbe026c4.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='testSplitTableRegionUnevenDaughter', {NAME => 'cf1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'cf2', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp 2023-04-12 02:53:54,797 DEBUG [RegionOpenAndInit-testSplitTableRegionUnevenDaughter-pool-0] regionserver.HRegion(866): Instantiated testSplitTableRegionUnevenDaughter,,1681268034745.cc9012ec0b656baa24d04368fbe026c4.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-12 02:53:54,797 DEBUG [RegionOpenAndInit-testSplitTableRegionUnevenDaughter-pool-0] regionserver.HRegion(1604): Closing cc9012ec0b656baa24d04368fbe026c4, disabling compactions & flushes 2023-04-12 02:53:54,797 INFO [RegionOpenAndInit-testSplitTableRegionUnevenDaughter-pool-0] regionserver.HRegion(1626): Closing region testSplitTableRegionUnevenDaughter,,1681268034745.cc9012ec0b656baa24d04368fbe026c4. 2023-04-12 02:53:54,797 DEBUG [RegionOpenAndInit-testSplitTableRegionUnevenDaughter-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testSplitTableRegionUnevenDaughter,,1681268034745.cc9012ec0b656baa24d04368fbe026c4. 2023-04-12 02:53:54,798 DEBUG [RegionOpenAndInit-testSplitTableRegionUnevenDaughter-pool-0] regionserver.HRegion(1714): Acquired close lock on testSplitTableRegionUnevenDaughter,,1681268034745.cc9012ec0b656baa24d04368fbe026c4. after waiting 0 ms 2023-04-12 02:53:54,798 DEBUG [RegionOpenAndInit-testSplitTableRegionUnevenDaughter-pool-0] regionserver.HRegion(1724): Updates disabled for region testSplitTableRegionUnevenDaughter,,1681268034745.cc9012ec0b656baa24d04368fbe026c4. 2023-04-12 02:53:54,798 INFO [RegionOpenAndInit-testSplitTableRegionUnevenDaughter-pool-0] regionserver.HRegion(1838): Closed testSplitTableRegionUnevenDaughter,,1681268034745.cc9012ec0b656baa24d04368fbe026c4. 2023-04-12 02:53:54,798 DEBUG [RegionOpenAndInit-testSplitTableRegionUnevenDaughter-pool-0] regionserver.HRegion(1558): Region close journal for cc9012ec0b656baa24d04368fbe026c4: 2023-04-12 02:53:54,802 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=testSplitTableRegionUnevenDaughter execute state=CREATE_TABLE_ADD_TO_META 2023-04-12 02:53:54,803 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testSplitTableRegionUnevenDaughter,,1681268034745.cc9012ec0b656baa24d04368fbe026c4.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1681268034803"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681268034803"}]},"ts":"1681268034803"} 2023-04-12 02:53:54,806 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-12 02:53:54,807 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=testSplitTableRegionUnevenDaughter execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-12 02:53:54,807 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testSplitTableRegionUnevenDaughter","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681268034807"}]},"ts":"1681268034807"} 2023-04-12 02:53:54,809 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testSplitTableRegionUnevenDaughter, state=ENABLING in hbase:meta 2023-04-12 02:53:54,814 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-04-12 02:53:54,816 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-04-12 02:53:54,816 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-04-12 02:53:54,816 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-04-12 02:53:54,816 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-04-12 02:53:54,816 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSplitTableRegionUnevenDaughter, region=cc9012ec0b656baa24d04368fbe026c4, ASSIGN}] 2023-04-12 02:53:54,819 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSplitTableRegionUnevenDaughter, region=cc9012ec0b656baa24d04368fbe026c4, ASSIGN 2023-04-12 02:53:54,821 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testSplitTableRegionUnevenDaughter, region=cc9012ec0b656baa24d04368fbe026c4, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,44679,1681268031772; forceNewPlan=false, retain=false 2023-04-12 02:53:54,971 INFO [jenkins-hbase4:40281] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-04-12 02:53:54,972 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=cc9012ec0b656baa24d04368fbe026c4, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:53:54,972 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitTableRegionUnevenDaughter,,1681268034745.cc9012ec0b656baa24d04368fbe026c4.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1681268034972"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681268034972"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681268034972"}]},"ts":"1681268034972"} 2023-04-12 02:53:54,976 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=11, ppid=10, state=RUNNABLE; OpenRegionProcedure cc9012ec0b656baa24d04368fbe026c4, server=jenkins-hbase4.apache.org,44679,1681268031772}] 2023-04-12 02:53:55,136 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testSplitTableRegionUnevenDaughter,,1681268034745.cc9012ec0b656baa24d04368fbe026c4. 2023-04-12 02:53:55,136 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => cc9012ec0b656baa24d04368fbe026c4, NAME => 'testSplitTableRegionUnevenDaughter,,1681268034745.cc9012ec0b656baa24d04368fbe026c4.', STARTKEY => '', ENDKEY => ''} 2023-04-12 02:53:55,136 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(174): System coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro loaded, priority=536870911. 2023-04-12 02:53:55,137 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testSplitTableRegionUnevenDaughter cc9012ec0b656baa24d04368fbe026c4 2023-04-12 02:53:55,137 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated testSplitTableRegionUnevenDaughter,,1681268034745.cc9012ec0b656baa24d04368fbe026c4.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-12 02:53:55,137 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for cc9012ec0b656baa24d04368fbe026c4 2023-04-12 02:53:55,137 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for cc9012ec0b656baa24d04368fbe026c4 2023-04-12 02:53:55,139 INFO [StoreOpener-cc9012ec0b656baa24d04368fbe026c4-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf1 of region cc9012ec0b656baa24d04368fbe026c4 2023-04-12 02:53:55,141 DEBUG [StoreOpener-cc9012ec0b656baa24d04368fbe026c4-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4/cf1 2023-04-12 02:53:55,141 DEBUG [StoreOpener-cc9012ec0b656baa24d04368fbe026c4-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4/cf1 2023-04-12 02:53:55,142 INFO [StoreOpener-cc9012ec0b656baa24d04368fbe026c4-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region cc9012ec0b656baa24d04368fbe026c4 columnFamilyName cf1 2023-04-12 02:53:55,143 INFO [StoreOpener-cc9012ec0b656baa24d04368fbe026c4-1] regionserver.HStore(310): Store=cc9012ec0b656baa24d04368fbe026c4/cf1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-12 02:53:55,143 INFO [StoreOpener-cc9012ec0b656baa24d04368fbe026c4-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf2 of region cc9012ec0b656baa24d04368fbe026c4 2023-04-12 02:53:55,145 DEBUG [StoreOpener-cc9012ec0b656baa24d04368fbe026c4-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4/cf2 2023-04-12 02:53:55,145 DEBUG [StoreOpener-cc9012ec0b656baa24d04368fbe026c4-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4/cf2 2023-04-12 02:53:55,146 INFO [StoreOpener-cc9012ec0b656baa24d04368fbe026c4-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region cc9012ec0b656baa24d04368fbe026c4 columnFamilyName cf2 2023-04-12 02:53:55,147 INFO [StoreOpener-cc9012ec0b656baa24d04368fbe026c4-1] regionserver.HStore(310): Store=cc9012ec0b656baa24d04368fbe026c4/cf2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-12 02:53:55,153 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4 2023-04-12 02:53:55,154 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4 2023-04-12 02:53:55,159 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testSplitTableRegionUnevenDaughter descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-04-12 02:53:55,161 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for cc9012ec0b656baa24d04368fbe026c4 2023-04-12 02:53:55,176 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-12 02:53:55,177 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened cc9012ec0b656baa24d04368fbe026c4; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=59790098, jitterRate=-0.10905811190605164}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-04-12 02:53:55,177 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for cc9012ec0b656baa24d04368fbe026c4: 2023-04-12 02:53:55,179 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testSplitTableRegionUnevenDaughter,,1681268034745.cc9012ec0b656baa24d04368fbe026c4., pid=11, masterSystemTime=1681268035129 2023-04-12 02:53:55,182 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testSplitTableRegionUnevenDaughter,,1681268034745.cc9012ec0b656baa24d04368fbe026c4. 2023-04-12 02:53:55,182 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testSplitTableRegionUnevenDaughter,,1681268034745.cc9012ec0b656baa24d04368fbe026c4. 2023-04-12 02:53:55,183 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=cc9012ec0b656baa24d04368fbe026c4, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:53:55,184 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testSplitTableRegionUnevenDaughter,,1681268034745.cc9012ec0b656baa24d04368fbe026c4.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1681268035183"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1681268035183"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681268035183"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681268035183"}]},"ts":"1681268035183"} 2023-04-12 02:53:55,192 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=11, resume processing ppid=10 2023-04-12 02:53:55,192 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=11, ppid=10, state=SUCCESS; OpenRegionProcedure cc9012ec0b656baa24d04368fbe026c4, server=jenkins-hbase4.apache.org,44679,1681268031772 in 212 msec 2023-04-12 02:53:55,197 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=10, resume processing ppid=9 2023-04-12 02:53:55,197 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=10, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=testSplitTableRegionUnevenDaughter, region=cc9012ec0b656baa24d04368fbe026c4, ASSIGN in 376 msec 2023-04-12 02:53:55,198 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=testSplitTableRegionUnevenDaughter execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-12 02:53:55,199 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testSplitTableRegionUnevenDaughter","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681268035198"}]},"ts":"1681268035198"} 2023-04-12 02:53:55,201 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testSplitTableRegionUnevenDaughter, state=ENABLED in hbase:meta 2023-04-12 02:53:55,205 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=testSplitTableRegionUnevenDaughter execute state=CREATE_TABLE_POST_OPERATION 2023-04-12 02:53:55,208 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter in 460 msec 2023-04-12 02:53:55,272 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40281] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 02:53:55,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40281] procedure.ProcedureCoordinator(165): Submitting procedure testSplitTableRegionUnevenDaughter 2023-04-12 02:53:55,281 INFO [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testSplitTableRegionUnevenDaughter' 2023-04-12 02:53:55,281 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:53:55,281 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testSplitTableRegionUnevenDaughter' starting 'acquire' 2023-04-12 02:53:55,281 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testSplitTableRegionUnevenDaughter', kicking off acquire phase on members. 2023-04-12 02:53:55,282 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:55,282 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:55,286 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:53:55,286 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:53:55,287 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:53:55,286 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:53:55,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:53:55,287 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:53:55,286 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:53:55,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:53:55,287 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:53:55,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:53:55,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:55,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:55,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:55,287 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:53:55,288 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 02:53:55,288 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:55,288 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:55,288 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:55,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 02:53:55,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 02:53:55,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:55,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:55,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 02:53:55,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:55,289 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionUnevenDaughter 2023-04-12 02:53:55,288 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionUnevenDaughter 2023-04-12 02:53:55,289 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionUnevenDaughter 2023-04-12 02:53:55,290 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionUnevenDaughter 2023-04-12 02:53:55,290 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionUnevenDaughter 2023-04-12 02:53:55,290 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionUnevenDaughter 2023-04-12 02:53:55,291 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionUnevenDaughter' with timeout 60000ms 2023-04-12 02:53:55,291 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionUnevenDaughter' with timeout 60000ms 2023-04-12 02:53:55,291 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:53:55,291 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:53:55,291 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionUnevenDaughter' with timeout 60000ms 2023-04-12 02:53:55,292 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionUnevenDaughter' starting 'acquire' stage 2023-04-12 02:53:55,291 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionUnevenDaughter' starting 'acquire' stage 2023-04-12 02:53:55,292 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionUnevenDaughter' locally acquired 2023-04-12 02:53:55,292 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionUnevenDaughter' locally acquired 2023-04-12 02:53:55,292 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:53:55,292 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43091,1681268031804' joining acquired barrier for procedure (testSplitTableRegionUnevenDaughter) in zk 2023-04-12 02:53:55,292 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionUnevenDaughter' starting 'acquire' stage 2023-04-12 02:53:55,292 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34579,1681268031719' joining acquired barrier for procedure (testSplitTableRegionUnevenDaughter) in zk 2023-04-12 02:53:55,293 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 02:53:55,294 DEBUG [rs(jenkins-hbase4.apache.org,44679,1681268031772)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testSplitTableRegionUnevenDaughter,,1681268034745.cc9012ec0b656baa24d04368fbe026c4. 2023-04-12 02:53:55,294 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 02:53:55,294 DEBUG [rs(jenkins-hbase4.apache.org,44679,1681268031772)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testSplitTableRegionUnevenDaughter,,1681268034745.cc9012ec0b656baa24d04368fbe026c4. started... 2023-04-12 02:53:55,295 INFO [rs(jenkins-hbase4.apache.org,44679,1681268031772)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing cc9012ec0b656baa24d04368fbe026c4 2/2 column families, dataSize=120 B heapSize=928 B 2023-04-12 02:53:55,295 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:55,296 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:55,296 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:55,296 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionUnevenDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 02:53:55,296 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:55,296 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionUnevenDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 02:53:55,376 INFO [rs(jenkins-hbase4.apache.org,44679,1681268031772)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=60 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4/.tmp/cf1/b2cffa1e1e9f40ba81d73ca9452dfeb4 2023-04-12 02:53:55,447 INFO [rs(jenkins-hbase4.apache.org,44679,1681268031772)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=60 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4/.tmp/cf2/18c4c7a41a6d44f9a7eb1d9d77806e17 2023-04-12 02:53:55,464 DEBUG [rs(jenkins-hbase4.apache.org,44679,1681268031772)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4/.tmp/cf1/b2cffa1e1e9f40ba81d73ca9452dfeb4 as hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4/cf1/b2cffa1e1e9f40ba81d73ca9452dfeb4 2023-04-12 02:53:55,476 INFO [rs(jenkins-hbase4.apache.org,44679,1681268031772)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4/cf1/b2cffa1e1e9f40ba81d73ca9452dfeb4, entries=2, sequenceid=6, filesize=4.8 K 2023-04-12 02:53:55,479 DEBUG [rs(jenkins-hbase4.apache.org,44679,1681268031772)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4/.tmp/cf2/18c4c7a41a6d44f9a7eb1d9d77806e17 as hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4/cf2/18c4c7a41a6d44f9a7eb1d9d77806e17 2023-04-12 02:53:55,489 INFO [rs(jenkins-hbase4.apache.org,44679,1681268031772)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4/cf2/18c4c7a41a6d44f9a7eb1d9d77806e17, entries=2, sequenceid=6, filesize=4.8 K 2023-04-12 02:53:55,491 INFO [rs(jenkins-hbase4.apache.org,44679,1681268031772)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~120 B/120, heapSize ~896 B/896, currentSize=0 B/0 for cc9012ec0b656baa24d04368fbe026c4 in 196ms, sequenceid=6, compaction requested=false 2023-04-12 02:53:55,493 DEBUG [rs(jenkins-hbase4.apache.org,44679,1681268031772)-flush-proc-pool-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'testSplitTableRegionUnevenDaughter' 2023-04-12 02:53:55,496 DEBUG [rs(jenkins-hbase4.apache.org,44679,1681268031772)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for cc9012ec0b656baa24d04368fbe026c4: 2023-04-12 02:53:55,496 DEBUG [rs(jenkins-hbase4.apache.org,44679,1681268031772)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testSplitTableRegionUnevenDaughter,,1681268034745.cc9012ec0b656baa24d04368fbe026c4. 2023-04-12 02:53:55,496 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 02:53:55,497 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 02:53:55,497 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:53:55,497 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionUnevenDaughter' locally acquired 2023-04-12 02:53:55,497 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44679,1681268031772' joining acquired barrier for procedure (testSplitTableRegionUnevenDaughter) in zk 2023-04-12 02:53:55,499 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:55,499 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:53:55,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:53:55,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 02:53:55,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 02:53:55,499 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:55,499 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionUnevenDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 02:53:55,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 02:53:55,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 02:53:55,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionUnevenDaughter 2023-04-12 02:53:55,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:53:55,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:53:55,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:53:55,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 02:53:55,502 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44679,1681268031772' joining acquired barrier for procedure 'testSplitTableRegionUnevenDaughter' on coordinator 2023-04-12 02:53:55,502 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testSplitTableRegionUnevenDaughter' starting 'in-barrier' execution. 2023-04-12 02:53:55,502 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3e9d0fc4[Count = 0] remaining members to acquire global barrier 2023-04-12 02:53:55,503 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:55,508 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:55,508 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:55,508 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:55,508 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:55,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:55,508 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:55,509 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionUnevenDaughter' received 'reached' from coordinator. 2023-04-12 02:53:55,509 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:53:55,509 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:55,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:55,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:55,509 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionUnevenDaughter' received 'reached' from coordinator. 2023-04-12 02:53:55,509 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionUnevenDaughter' received 'reached' from coordinator. 2023-04-12 02:53:55,509 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionUnevenDaughter' locally completed 2023-04-12 02:53:55,509 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionUnevenDaughter' locally completed 2023-04-12 02:53:55,509 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 02:53:55,509 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionUnevenDaughter' locally completed 2023-04-12 02:53:55,509 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionUnevenDaughter' completed for member 'jenkins-hbase4.apache.org,43091,1681268031804' in zk 2023-04-12 02:53:55,509 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionUnevenDaughter' completed for member 'jenkins-hbase4.apache.org,34579,1681268031719' in zk 2023-04-12 02:53:55,509 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionUnevenDaughter' completed for member 'jenkins-hbase4.apache.org,44679,1681268031772' in zk 2023-04-12 02:53:55,513 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionUnevenDaughter' has notified controller of completion 2023-04-12 02:53:55,513 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:53:55,513 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:53:55,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:53:55,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 02:53:55,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 02:53:55,513 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionUnevenDaughter' has notified controller of completion 2023-04-12 02:53:55,513 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionUnevenDaughter' completed. 2023-04-12 02:53:55,514 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionUnevenDaughter' has notified controller of completion 2023-04-12 02:53:55,514 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:53:55,514 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:53:55,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 02:53:55,514 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionUnevenDaughter' completed. 2023-04-12 02:53:55,515 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionUnevenDaughter' completed. 2023-04-12 02:53:55,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 02:53:55,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionUnevenDaughter 2023-04-12 02:53:55,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:53:55,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:53:55,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:53:55,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 02:53:55,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionUnevenDaughter 2023-04-12 02:53:55,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:53:55,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:53:55,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:53:55,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testSplitTableRegionUnevenDaughter' member 'jenkins-hbase4.apache.org,44679,1681268031772': 2023-04-12 02:53:55,520 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44679,1681268031772' released barrier for procedure'testSplitTableRegionUnevenDaughter', counting down latch. Waiting for 0 more 2023-04-12 02:53:55,520 INFO [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testSplitTableRegionUnevenDaughter' execution completed 2023-04-12 02:53:55,520 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 02:53:55,520 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 02:53:55,520 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testSplitTableRegionUnevenDaughter 2023-04-12 02:53:55,520 INFO [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testSplitTableRegionUnevenDaughterincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 02:53:55,523 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:55,523 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:55,523 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:55,523 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:53:55,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:55,523 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:55,523 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:55,523 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:53:55,523 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:55,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:55,523 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:55,523 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:53:55,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:55,524 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:53:55,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:55,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 02:53:55,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 02:53:55,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:53:55,524 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:53:55,524 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:53:55,524 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:53:55,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:53:55,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 02:53:55,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:53:55,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:55,525 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:53:55,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:55,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionUnevenDaughter 2023-04-12 02:53:55,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:55,525 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:53:55,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 02:53:55,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionUnevenDaughter 2023-04-12 02:53:55,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:53:55,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:53:55,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:53:55,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 02:53:55,528 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:53:55,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionUnevenDaughter 2023-04-12 02:53:55,528 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:53:55,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:53:55,529 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:53:55,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:53:55,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:53:55,540 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:53:55,540 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:53:55,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:53:55,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:53:55,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:53:55,540 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:53:55,540 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:53:55,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:53:55,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:53:55,541 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:55,541 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:53:55,541 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:53:55,541 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40281] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testSplitTableRegionUnevenDaughter' 2023-04-12 02:53:55,541 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:53:55,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:53:55,541 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:53:55,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:53:55,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:53:55,541 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:53:55,541 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40281] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 02:53:55,541 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:53:55,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:53:55,542 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:55,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:53:55,542 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:53:55,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:53:55,543 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:55,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:53:55,543 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:53:55,543 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:53:55,543 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:55,543 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:55,544 DEBUG [Listener at localhost/46667] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testSplitTableRegionUnevenDaughter'' to complete. (max 20000 ms per retry) 2023-04-12 02:53:55,544 DEBUG [Listener at localhost/46667] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 02:53:55,644 DEBUG [Listener at localhost/46667] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 02:53:55,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40281] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 02:53:55,673 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40281] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 02:53:55,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40281] procedure.ProcedureCoordinator(143): Procedure testSplitTableRegionUnevenDaughter was in running list but was completed. Accepting new attempt. 2023-04-12 02:53:55,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40281] procedure.ProcedureCoordinator(165): Submitting procedure testSplitTableRegionUnevenDaughter 2023-04-12 02:53:55,676 INFO [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testSplitTableRegionUnevenDaughter' 2023-04-12 02:53:55,677 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:53:55,677 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testSplitTableRegionUnevenDaughter' starting 'acquire' 2023-04-12 02:53:55,677 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testSplitTableRegionUnevenDaughter', kicking off acquire phase on members. 2023-04-12 02:53:55,678 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:55,678 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:55,679 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:53:55,679 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:53:55,679 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:53:55,679 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:53:55,679 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:53:55,680 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:53:55,680 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:53:55,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:53:55,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:53:55,680 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:53:55,680 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 02:53:55,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:53:55,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:55,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:55,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:55,681 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:55,681 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:55,681 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:55,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 02:53:55,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:55,681 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionUnevenDaughter 2023-04-12 02:53:55,682 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionUnevenDaughter is still present, removing 2023-04-12 02:53:55,682 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionUnevenDaughter 2023-04-12 02:53:55,682 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionUnevenDaughter' with timeout 60000ms 2023-04-12 02:53:55,682 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:53:55,683 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionUnevenDaughter' starting 'acquire' stage 2023-04-12 02:53:55,683 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionUnevenDaughter' locally acquired 2023-04-12 02:53:55,683 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43091,1681268031804' joining acquired barrier for procedure (testSplitTableRegionUnevenDaughter) in zk 2023-04-12 02:53:55,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 02:53:55,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 02:53:55,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:55,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:55,684 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionUnevenDaughter 2023-04-12 02:53:55,684 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionUnevenDaughter 2023-04-12 02:53:55,684 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionUnevenDaughter is still present, removing 2023-04-12 02:53:55,685 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionUnevenDaughter 2023-04-12 02:53:55,685 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionUnevenDaughter' with timeout 60000ms 2023-04-12 02:53:55,685 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionUnevenDaughter is still present, removing 2023-04-12 02:53:55,685 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:53:55,685 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionUnevenDaughter 2023-04-12 02:53:55,685 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionUnevenDaughter' starting 'acquire' stage 2023-04-12 02:53:55,685 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionUnevenDaughter' with timeout 60000ms 2023-04-12 02:53:55,685 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:53:55,685 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionUnevenDaughter' locally acquired 2023-04-12 02:53:55,686 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionUnevenDaughter' starting 'acquire' stage 2023-04-12 02:53:55,686 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:55,686 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34579,1681268031719' joining acquired barrier for procedure (testSplitTableRegionUnevenDaughter) in zk 2023-04-12 02:53:55,686 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:55,687 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionUnevenDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 02:53:55,687 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 02:53:55,688 DEBUG [rs(jenkins-hbase4.apache.org,44679,1681268031772)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testSplitTableRegionUnevenDaughter,,1681268034745.cc9012ec0b656baa24d04368fbe026c4. 2023-04-12 02:53:55,688 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 02:53:55,688 DEBUG [rs(jenkins-hbase4.apache.org,44679,1681268031772)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testSplitTableRegionUnevenDaughter,,1681268034745.cc9012ec0b656baa24d04368fbe026c4. started... 2023-04-12 02:53:55,688 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:55,688 INFO [rs(jenkins-hbase4.apache.org,44679,1681268031772)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing cc9012ec0b656baa24d04368fbe026c4 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-04-12 02:53:55,688 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:55,688 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionUnevenDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 02:53:55,708 INFO [rs(jenkins-hbase4.apache.org,44679,1681268031772)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=19 (bloomFilter=true), to=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4/.tmp/cf1/07c3d7abf05f43cdbe44fe637da34a7d 2023-04-12 02:53:55,735 INFO [rs(jenkins-hbase4.apache.org,44679,1681268031772)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=19 (bloomFilter=true), to=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4/.tmp/cf2/8df43cbe10c74ad8b2c721f3509bb1d1 2023-04-12 02:53:55,744 DEBUG [rs(jenkins-hbase4.apache.org,44679,1681268031772)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4/.tmp/cf1/07c3d7abf05f43cdbe44fe637da34a7d as hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4/cf1/07c3d7abf05f43cdbe44fe637da34a7d 2023-04-12 02:53:55,752 INFO [rs(jenkins-hbase4.apache.org,44679,1681268031772)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4/cf1/07c3d7abf05f43cdbe44fe637da34a7d, entries=10, sequenceid=19, filesize=5.0 K 2023-04-12 02:53:55,754 DEBUG [rs(jenkins-hbase4.apache.org,44679,1681268031772)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4/.tmp/cf2/8df43cbe10c74ad8b2c721f3509bb1d1 as hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4/cf2/8df43cbe10c74ad8b2c721f3509bb1d1 2023-04-12 02:53:55,761 INFO [rs(jenkins-hbase4.apache.org,44679,1681268031772)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4/cf2/8df43cbe10c74ad8b2c721f3509bb1d1, entries=10, sequenceid=19, filesize=5.0 K 2023-04-12 02:53:55,763 INFO [rs(jenkins-hbase4.apache.org,44679,1681268031772)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for cc9012ec0b656baa24d04368fbe026c4 in 75ms, sequenceid=19, compaction requested=false 2023-04-12 02:53:55,763 DEBUG [rs(jenkins-hbase4.apache.org,44679,1681268031772)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for cc9012ec0b656baa24d04368fbe026c4: 2023-04-12 02:53:55,763 DEBUG [rs(jenkins-hbase4.apache.org,44679,1681268031772)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testSplitTableRegionUnevenDaughter,,1681268034745.cc9012ec0b656baa24d04368fbe026c4. 2023-04-12 02:53:55,763 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 02:53:55,764 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 02:53:55,764 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:53:55,764 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionUnevenDaughter' locally acquired 2023-04-12 02:53:55,764 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44679,1681268031772' joining acquired barrier for procedure (testSplitTableRegionUnevenDaughter) in zk 2023-04-12 02:53:55,767 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:53:55,767 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:55,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:53:55,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 02:53:55,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 02:53:55,767 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:55,767 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionUnevenDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 02:53:55,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 02:53:55,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 02:53:55,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionUnevenDaughter 2023-04-12 02:53:55,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:53:55,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:53:55,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:53:55,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 02:53:55,770 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44679,1681268031772' joining acquired barrier for procedure 'testSplitTableRegionUnevenDaughter' on coordinator 2023-04-12 02:53:55,770 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@c3c378b[Count = 0] remaining members to acquire global barrier 2023-04-12 02:53:55,770 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testSplitTableRegionUnevenDaughter' starting 'in-barrier' execution. 2023-04-12 02:53:55,770 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:55,771 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:55,771 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:55,771 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:55,771 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:55,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:55,771 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:55,771 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionUnevenDaughter' received 'reached' from coordinator. 2023-04-12 02:53:55,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:55,771 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:53:55,772 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 02:53:55,771 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:55,772 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionUnevenDaughter' received 'reached' from coordinator. 2023-04-12 02:53:55,771 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionUnevenDaughter' locally completed 2023-04-12 02:53:55,772 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionUnevenDaughter' locally completed 2023-04-12 02:53:55,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:55,772 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionUnevenDaughter' completed for member 'jenkins-hbase4.apache.org,44679,1681268031772' in zk 2023-04-12 02:53:55,772 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionUnevenDaughter' received 'reached' from coordinator. 2023-04-12 02:53:55,772 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionUnevenDaughter' locally completed 2023-04-12 02:53:55,772 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionUnevenDaughter' completed for member 'jenkins-hbase4.apache.org,43091,1681268031804' in zk 2023-04-12 02:53:55,772 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionUnevenDaughter' completed for member 'jenkins-hbase4.apache.org,34579,1681268031719' in zk 2023-04-12 02:53:55,774 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionUnevenDaughter' has notified controller of completion 2023-04-12 02:53:55,774 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:53:55,774 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:53:55,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:53:55,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 02:53:55,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 02:53:55,774 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionUnevenDaughter' completed. 2023-04-12 02:53:55,775 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionUnevenDaughter' has notified controller of completion 2023-04-12 02:53:55,775 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:53:55,775 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionUnevenDaughter' has notified controller of completion 2023-04-12 02:53:55,775 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionUnevenDaughter' completed. 2023-04-12 02:53:55,775 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:53:55,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 02:53:55,776 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionUnevenDaughter' completed. 2023-04-12 02:53:55,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 02:53:55,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionUnevenDaughter 2023-04-12 02:53:55,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:53:55,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:53:55,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:53:55,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 02:53:55,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionUnevenDaughter 2023-04-12 02:53:55,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:53:55,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:53:55,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:53:55,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testSplitTableRegionUnevenDaughter' member 'jenkins-hbase4.apache.org,44679,1681268031772': 2023-04-12 02:53:55,780 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44679,1681268031772' released barrier for procedure'testSplitTableRegionUnevenDaughter', counting down latch. Waiting for 0 more 2023-04-12 02:53:55,780 INFO [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testSplitTableRegionUnevenDaughter' execution completed 2023-04-12 02:53:55,780 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 02:53:55,780 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 02:53:55,781 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testSplitTableRegionUnevenDaughter 2023-04-12 02:53:55,781 INFO [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testSplitTableRegionUnevenDaughterincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 02:53:55,783 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:55,783 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:55,783 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:55,783 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:55,783 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:53:55,783 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:53:55,783 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:55,783 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:55,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:55,784 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:55,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 02:53:55,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 02:53:55,784 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:53:55,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:55,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:55,784 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:53:55,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:55,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 02:53:55,785 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:53:55,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:53:55,785 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:53:55,785 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:53:55,785 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:53:55,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:53:55,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionUnevenDaughter 2023-04-12 02:53:55,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:53:55,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:55,785 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:53:55,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:55,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:55,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 02:53:55,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionUnevenDaughter 2023-04-12 02:53:55,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:53:55,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:53:55,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:53:55,788 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:53:55,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 02:53:55,788 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:53:55,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionUnevenDaughter 2023-04-12 02:53:55,788 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:53:55,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:53:55,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:53:55,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:53:55,798 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:53:55,798 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:53:55,798 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:55,798 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:53:55,798 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:53:55,798 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:53:55,798 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:53:55,798 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40281] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testSplitTableRegionUnevenDaughter' 2023-04-12 02:53:55,799 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40281] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 02:53:55,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:53:55,798 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:53:55,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:53:55,798 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:53:55,798 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:53:55,798 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:53:55,799 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:53:55,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:53:55,799 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:53:55,799 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:53:55,799 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:53:55,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:53:55,799 DEBUG [Listener at localhost/46667] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testSplitTableRegionUnevenDaughter'' to complete. (max 20000 ms per retry) 2023-04-12 02:53:55,799 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:53:55,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:53:55,799 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:53:55,799 DEBUG [Listener at localhost/46667] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 02:53:55,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:53:55,800 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:55,800 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:53:55,800 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:55,800 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:53:55,800 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:53:55,800 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:55,800 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:55,900 DEBUG [Listener at localhost/46667] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 02:53:55,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40281] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 02:53:55,929 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40281] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 02:53:55,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40281] procedure.ProcedureCoordinator(143): Procedure testSplitTableRegionUnevenDaughter was in running list but was completed. Accepting new attempt. 2023-04-12 02:53:55,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40281] procedure.ProcedureCoordinator(165): Submitting procedure testSplitTableRegionUnevenDaughter 2023-04-12 02:53:55,932 INFO [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testSplitTableRegionUnevenDaughter' 2023-04-12 02:53:55,932 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:53:55,933 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testSplitTableRegionUnevenDaughter' starting 'acquire' 2023-04-12 02:53:55,933 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testSplitTableRegionUnevenDaughter', kicking off acquire phase on members. 2023-04-12 02:53:55,933 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:55,933 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:55,935 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:53:55,935 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:53:55,935 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:53:55,935 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:53:55,935 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:53:55,935 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:53:55,935 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:53:55,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:53:55,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:53:55,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:53:55,936 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:53:55,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:55,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:55,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:55,936 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 02:53:55,936 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:55,936 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:55,936 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:55,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 02:53:55,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:55,937 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionUnevenDaughter 2023-04-12 02:53:55,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 02:53:55,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 02:53:55,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:55,937 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionUnevenDaughter is still present, removing 2023-04-12 02:53:55,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:55,937 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionUnevenDaughter 2023-04-12 02:53:55,937 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionUnevenDaughter 2023-04-12 02:53:55,939 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionUnevenDaughter' with timeout 60000ms 2023-04-12 02:53:55,937 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionUnevenDaughter 2023-04-12 02:53:55,939 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:53:55,939 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionUnevenDaughter is still present, removing 2023-04-12 02:53:55,940 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionUnevenDaughter is still present, removing 2023-04-12 02:53:55,940 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionUnevenDaughter' starting 'acquire' stage 2023-04-12 02:53:55,940 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionUnevenDaughter 2023-04-12 02:53:55,940 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionUnevenDaughter 2023-04-12 02:53:55,940 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionUnevenDaughter' with timeout 60000ms 2023-04-12 02:53:55,940 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionUnevenDaughter' with timeout 60000ms 2023-04-12 02:53:55,940 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:53:55,940 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:53:55,941 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionUnevenDaughter' starting 'acquire' stage 2023-04-12 02:53:55,940 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionUnevenDaughter' locally acquired 2023-04-12 02:53:55,941 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionUnevenDaughter' starting 'acquire' stage 2023-04-12 02:53:55,942 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionUnevenDaughter' locally acquired 2023-04-12 02:53:55,942 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43091,1681268031804' joining acquired barrier for procedure (testSplitTableRegionUnevenDaughter) in zk 2023-04-12 02:53:55,941 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34579,1681268031719' joining acquired barrier for procedure (testSplitTableRegionUnevenDaughter) in zk 2023-04-12 02:53:55,942 DEBUG [rs(jenkins-hbase4.apache.org,44679,1681268031772)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testSplitTableRegionUnevenDaughter,,1681268034745.cc9012ec0b656baa24d04368fbe026c4. 2023-04-12 02:53:55,942 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 02:53:55,942 DEBUG [rs(jenkins-hbase4.apache.org,44679,1681268031772)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testSplitTableRegionUnevenDaughter,,1681268034745.cc9012ec0b656baa24d04368fbe026c4. started... 2023-04-12 02:53:55,942 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 02:53:55,942 INFO [rs(jenkins-hbase4.apache.org,44679,1681268031772)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing cc9012ec0b656baa24d04368fbe026c4 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-04-12 02:53:55,945 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:55,945 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:55,945 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:55,946 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionUnevenDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 02:53:55,946 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:55,946 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionUnevenDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 02:53:55,962 INFO [rs(jenkins-hbase4.apache.org,44679,1681268031772)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=32 (bloomFilter=true), to=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4/.tmp/cf1/364127255e644c3d8f0a55272e1ab83a 2023-04-12 02:53:55,986 INFO [rs(jenkins-hbase4.apache.org,44679,1681268031772)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=32 (bloomFilter=true), to=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4/.tmp/cf2/1b7cac025fb940cab9783583b39cef46 2023-04-12 02:53:55,994 DEBUG [rs(jenkins-hbase4.apache.org,44679,1681268031772)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4/.tmp/cf1/364127255e644c3d8f0a55272e1ab83a as hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4/cf1/364127255e644c3d8f0a55272e1ab83a 2023-04-12 02:53:56,002 INFO [rs(jenkins-hbase4.apache.org,44679,1681268031772)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4/cf1/364127255e644c3d8f0a55272e1ab83a, entries=10, sequenceid=32, filesize=5.0 K 2023-04-12 02:53:56,003 DEBUG [rs(jenkins-hbase4.apache.org,44679,1681268031772)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4/.tmp/cf2/1b7cac025fb940cab9783583b39cef46 as hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4/cf2/1b7cac025fb940cab9783583b39cef46 2023-04-12 02:53:56,010 INFO [rs(jenkins-hbase4.apache.org,44679,1681268031772)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4/cf2/1b7cac025fb940cab9783583b39cef46, entries=10, sequenceid=32, filesize=5.0 K 2023-04-12 02:53:56,012 INFO [rs(jenkins-hbase4.apache.org,44679,1681268031772)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for cc9012ec0b656baa24d04368fbe026c4 in 70ms, sequenceid=32, compaction requested=true 2023-04-12 02:53:56,012 DEBUG [rs(jenkins-hbase4.apache.org,44679,1681268031772)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for cc9012ec0b656baa24d04368fbe026c4: 2023-04-12 02:53:56,012 DEBUG [rs(jenkins-hbase4.apache.org,44679,1681268031772)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testSplitTableRegionUnevenDaughter,,1681268034745.cc9012ec0b656baa24d04368fbe026c4. 2023-04-12 02:53:56,012 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 02:53:56,013 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 02:53:56,013 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:53:56,013 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionUnevenDaughter' locally acquired 2023-04-12 02:53:56,013 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44679,1681268031772' joining acquired barrier for procedure (testSplitTableRegionUnevenDaughter) in zk 2023-04-12 02:53:56,015 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,015 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:53:56,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:53:56,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 02:53:56,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 02:53:56,016 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 02:53:56,016 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionUnevenDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 02:53:56,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 02:53:56,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:53:56,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:53:56,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:53:56,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 02:53:56,018 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44679,1681268031772' joining acquired barrier for procedure 'testSplitTableRegionUnevenDaughter' on coordinator 2023-04-12 02:53:56,018 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@20c8844[Count = 0] remaining members to acquire global barrier 2023-04-12 02:53:56,018 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testSplitTableRegionUnevenDaughter' starting 'in-barrier' execution. 2023-04-12 02:53:56,018 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,020 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,020 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,020 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,020 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,020 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,020 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:53:56,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,020 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 02:53:56,020 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionUnevenDaughter' received 'reached' from coordinator. 2023-04-12 02:53:56,020 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionUnevenDaughter' locally completed 2023-04-12 02:53:56,020 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionUnevenDaughter' received 'reached' from coordinator. 2023-04-12 02:53:56,020 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionUnevenDaughter' locally completed 2023-04-12 02:53:56,020 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,020 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionUnevenDaughter' completed for member 'jenkins-hbase4.apache.org,34579,1681268031719' in zk 2023-04-12 02:53:56,020 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionUnevenDaughter' completed for member 'jenkins-hbase4.apache.org,43091,1681268031804' in zk 2023-04-12 02:53:56,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,021 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionUnevenDaughter' received 'reached' from coordinator. 2023-04-12 02:53:56,021 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionUnevenDaughter' locally completed 2023-04-12 02:53:56,021 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionUnevenDaughter' completed for member 'jenkins-hbase4.apache.org,44679,1681268031772' in zk 2023-04-12 02:53:56,025 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionUnevenDaughter' has notified controller of completion 2023-04-12 02:53:56,025 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:53:56,025 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionUnevenDaughter' completed. 2023-04-12 02:53:56,026 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionUnevenDaughter' has notified controller of completion 2023-04-12 02:53:56,026 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionUnevenDaughter' has notified controller of completion 2023-04-12 02:53:56,026 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:53:56,026 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:53:56,026 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionUnevenDaughter' completed. 2023-04-12 02:53:56,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:53:56,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 02:53:56,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 02:53:56,026 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:53:56,027 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionUnevenDaughter' completed. 2023-04-12 02:53:56,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 02:53:56,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 02:53:56,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:53:56,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:53:56,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:53:56,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 02:53:56,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:53:56,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:53:56,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:53:56,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testSplitTableRegionUnevenDaughter' member 'jenkins-hbase4.apache.org,44679,1681268031772': 2023-04-12 02:53:56,032 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44679,1681268031772' released barrier for procedure'testSplitTableRegionUnevenDaughter', counting down latch. Waiting for 0 more 2023-04-12 02:53:56,032 INFO [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testSplitTableRegionUnevenDaughter' execution completed 2023-04-12 02:53:56,032 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 02:53:56,032 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 02:53:56,032 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,032 INFO [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testSplitTableRegionUnevenDaughterincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 02:53:56,034 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,034 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,034 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,034 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,034 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:53:56,034 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:53:56,034 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,034 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,034 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,034 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:53:56,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 02:53:56,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 02:53:56,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,035 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:53:56,035 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:53:56,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:53:56,035 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:53:56,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:53:56,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 02:53:56,035 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:53:56,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:53:56,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,036 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:53:56,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,036 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:53:56,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 02:53:56,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:53:56,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:53:56,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:53:56,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 02:53:56,039 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:53:56,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,039 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:53:56,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:53:56,039 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:53:56,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:53:56,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:53:56,043 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:53:56,043 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:53:56,043 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:53:56,043 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:53:56,043 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,043 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:53:56,043 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:53:56,043 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:53:56,043 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:53:56,043 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:53:56,043 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:53:56,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:53:56,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:53:56,043 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40281] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testSplitTableRegionUnevenDaughter' 2023-04-12 02:53:56,044 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40281] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 02:53:56,043 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:53:56,044 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:53:56,044 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,044 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:53:56,043 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:53:56,044 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:53:56,044 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:53:56,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:53:56,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:53:56,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:53:56,044 DEBUG [Listener at localhost/46667] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testSplitTableRegionUnevenDaughter'' to complete. (max 20000 ms per retry) 2023-04-12 02:53:56,044 DEBUG [Listener at localhost/46667] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 02:53:56,044 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,044 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:53:56,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:53:56,045 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:53:56,045 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:53:56,045 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,045 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,145 DEBUG [Listener at localhost/46667] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 02:53:56,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40281] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 02:53:56,168 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40281] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 02:53:56,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40281] procedure.ProcedureCoordinator(143): Procedure testSplitTableRegionUnevenDaughter was in running list but was completed. Accepting new attempt. 2023-04-12 02:53:56,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40281] procedure.ProcedureCoordinator(165): Submitting procedure testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,170 INFO [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testSplitTableRegionUnevenDaughter' 2023-04-12 02:53:56,170 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:53:56,171 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testSplitTableRegionUnevenDaughter' starting 'acquire' 2023-04-12 02:53:56,171 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testSplitTableRegionUnevenDaughter', kicking off acquire phase on members. 2023-04-12 02:53:56,172 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,172 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,174 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:53:56,174 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:53:56,174 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:53:56,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:53:56,174 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:53:56,174 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:53:56,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:53:56,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:53:56,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:53:56,175 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:53:56,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:53:56,175 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 02:53:56,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,176 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,176 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,176 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 02:53:56,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 02:53:56,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 02:53:56,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,176 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,176 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,176 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,177 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionUnevenDaughter is still present, removing 2023-04-12 02:53:56,177 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,177 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionUnevenDaughter is still present, removing 2023-04-12 02:53:56,177 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionUnevenDaughter' with timeout 60000ms 2023-04-12 02:53:56,177 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,177 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:53:56,177 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionUnevenDaughter is still present, removing 2023-04-12 02:53:56,178 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionUnevenDaughter' starting 'acquire' stage 2023-04-12 02:53:56,177 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionUnevenDaughter' with timeout 60000ms 2023-04-12 02:53:56,178 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionUnevenDaughter' locally acquired 2023-04-12 02:53:56,178 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,178 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34579,1681268031719' joining acquired barrier for procedure (testSplitTableRegionUnevenDaughter) in zk 2023-04-12 02:53:56,178 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionUnevenDaughter' with timeout 60000ms 2023-04-12 02:53:56,178 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:53:56,178 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:53:56,179 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionUnevenDaughter' starting 'acquire' stage 2023-04-12 02:53:56,179 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionUnevenDaughter' starting 'acquire' stage 2023-04-12 02:53:56,179 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionUnevenDaughter' locally acquired 2023-04-12 02:53:56,180 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43091,1681268031804' joining acquired barrier for procedure (testSplitTableRegionUnevenDaughter) in zk 2023-04-12 02:53:56,180 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 02:53:56,180 DEBUG [rs(jenkins-hbase4.apache.org,44679,1681268031772)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testSplitTableRegionUnevenDaughter,,1681268034745.cc9012ec0b656baa24d04368fbe026c4. 2023-04-12 02:53:56,180 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 02:53:56,180 DEBUG [rs(jenkins-hbase4.apache.org,44679,1681268031772)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testSplitTableRegionUnevenDaughter,,1681268034745.cc9012ec0b656baa24d04368fbe026c4. started... 2023-04-12 02:53:56,180 INFO [rs(jenkins-hbase4.apache.org,44679,1681268031772)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing cc9012ec0b656baa24d04368fbe026c4 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-04-12 02:53:56,181 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,182 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,182 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,182 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionUnevenDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 02:53:56,182 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,182 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionUnevenDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 02:53:56,203 INFO [rs(jenkins-hbase4.apache.org,44679,1681268031772)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=45 (bloomFilter=true), to=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4/.tmp/cf1/a859f22056324f46889984d0e730b4bf 2023-04-12 02:53:56,231 INFO [rs(jenkins-hbase4.apache.org,44679,1681268031772)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=45 (bloomFilter=true), to=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4/.tmp/cf2/213df881d096427c9a5864569db2e1c4 2023-04-12 02:53:56,238 DEBUG [rs(jenkins-hbase4.apache.org,44679,1681268031772)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4/.tmp/cf1/a859f22056324f46889984d0e730b4bf as hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4/cf1/a859f22056324f46889984d0e730b4bf 2023-04-12 02:53:56,245 INFO [rs(jenkins-hbase4.apache.org,44679,1681268031772)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4/cf1/a859f22056324f46889984d0e730b4bf, entries=10, sequenceid=45, filesize=5.0 K 2023-04-12 02:53:56,246 DEBUG [rs(jenkins-hbase4.apache.org,44679,1681268031772)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4/.tmp/cf2/213df881d096427c9a5864569db2e1c4 as hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4/cf2/213df881d096427c9a5864569db2e1c4 2023-04-12 02:53:56,254 INFO [rs(jenkins-hbase4.apache.org,44679,1681268031772)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4/cf2/213df881d096427c9a5864569db2e1c4, entries=10, sequenceid=45, filesize=5.0 K 2023-04-12 02:53:56,256 INFO [rs(jenkins-hbase4.apache.org,44679,1681268031772)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for cc9012ec0b656baa24d04368fbe026c4 in 76ms, sequenceid=45, compaction requested=true 2023-04-12 02:53:56,256 DEBUG [rs(jenkins-hbase4.apache.org,44679,1681268031772)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for cc9012ec0b656baa24d04368fbe026c4: 2023-04-12 02:53:56,256 DEBUG [rs(jenkins-hbase4.apache.org,44679,1681268031772)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testSplitTableRegionUnevenDaughter,,1681268034745.cc9012ec0b656baa24d04368fbe026c4. 2023-04-12 02:53:56,256 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 02:53:56,257 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 02:53:56,257 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:53:56,257 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionUnevenDaughter' locally acquired 2023-04-12 02:53:56,257 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44679,1681268031772' joining acquired barrier for procedure (testSplitTableRegionUnevenDaughter) in zk 2023-04-12 02:53:56,268 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:53:56,268 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:53:56,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 02:53:56,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 02:53:56,268 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,269 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionUnevenDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 02:53:56,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 02:53:56,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 02:53:56,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:53:56,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:53:56,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:53:56,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 02:53:56,271 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44679,1681268031772' joining acquired barrier for procedure 'testSplitTableRegionUnevenDaughter' on coordinator 2023-04-12 02:53:56,271 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@39d81d82[Count = 0] remaining members to acquire global barrier 2023-04-12 02:53:56,271 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testSplitTableRegionUnevenDaughter' starting 'in-barrier' execution. 2023-04-12 02:53:56,271 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,273 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,273 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,273 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,273 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,273 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,273 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,273 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionUnevenDaughter' received 'reached' from coordinator. 2023-04-12 02:53:56,273 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionUnevenDaughter' received 'reached' from coordinator. 2023-04-12 02:53:56,273 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionUnevenDaughter' locally completed 2023-04-12 02:53:56,273 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:53:56,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,273 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 02:53:56,274 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionUnevenDaughter' received 'reached' from coordinator. 2023-04-12 02:53:56,273 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionUnevenDaughter' completed for member 'jenkins-hbase4.apache.org,44679,1681268031772' in zk 2023-04-12 02:53:56,273 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionUnevenDaughter' locally completed 2023-04-12 02:53:56,274 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionUnevenDaughter' locally completed 2023-04-12 02:53:56,274 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionUnevenDaughter' completed for member 'jenkins-hbase4.apache.org,43091,1681268031804' in zk 2023-04-12 02:53:56,274 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionUnevenDaughter' completed for member 'jenkins-hbase4.apache.org,34579,1681268031719' in zk 2023-04-12 02:53:56,275 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionUnevenDaughter' has notified controller of completion 2023-04-12 02:53:56,275 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:53:56,275 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:53:56,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:53:56,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 02:53:56,275 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionUnevenDaughter' completed. 2023-04-12 02:53:56,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 02:53:56,277 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionUnevenDaughter' has notified controller of completion 2023-04-12 02:53:56,277 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:53:56,277 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionUnevenDaughter' has notified controller of completion 2023-04-12 02:53:56,277 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:53:56,277 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionUnevenDaughter' completed. 2023-04-12 02:53:56,277 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionUnevenDaughter' completed. 2023-04-12 02:53:56,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 02:53:56,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 02:53:56,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:53:56,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:53:56,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:53:56,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 02:53:56,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:53:56,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:53:56,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:53:56,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testSplitTableRegionUnevenDaughter' member 'jenkins-hbase4.apache.org,44679,1681268031772': 2023-04-12 02:53:56,282 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44679,1681268031772' released barrier for procedure'testSplitTableRegionUnevenDaughter', counting down latch. Waiting for 0 more 2023-04-12 02:53:56,282 INFO [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testSplitTableRegionUnevenDaughter' execution completed 2023-04-12 02:53:56,282 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 02:53:56,282 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 02:53:56,282 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,282 INFO [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testSplitTableRegionUnevenDaughterincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 02:53:56,285 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,285 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,285 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,285 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:53:56,285 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,285 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,285 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:53:56,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 02:53:56,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 02:53:56,285 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,286 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:53:56,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,286 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,286 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:53:56,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 02:53:56,286 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:53:56,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:53:56,287 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:53:56,287 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:53:56,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:53:56,287 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:53:56,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:53:56,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,287 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:53:56,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 02:53:56,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:53:56,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:53:56,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:53:56,290 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:53:56,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 02:53:56,291 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:53:56,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,291 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:53:56,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:53:56,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:53:56,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:53:56,297 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:53:56,297 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:53:56,297 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:53:56,297 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,297 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:53:56,297 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:53:56,297 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,297 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:53:56,297 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,297 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:53:56,297 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40281] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testSplitTableRegionUnevenDaughter' 2023-04-12 02:53:56,298 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40281] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 02:53:56,298 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:53:56,298 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,298 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,298 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:53:56,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:53:56,298 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:53:56,298 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:53:56,297 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:53:56,298 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:53:56,298 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:53:56,299 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:53:56,299 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:53:56,298 DEBUG [Listener at localhost/46667] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testSplitTableRegionUnevenDaughter'' to complete. (max 20000 ms per retry) 2023-04-12 02:53:56,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:53:56,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:53:56,299 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:53:56,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:53:56,300 DEBUG [Listener at localhost/46667] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 02:53:56,301 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:53:56,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:53:56,301 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:53:56,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:53:56,401 DEBUG [Listener at localhost/46667] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 02:53:56,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40281] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 02:53:56,424 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40281] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 02:53:56,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40281] procedure.ProcedureCoordinator(143): Procedure testSplitTableRegionUnevenDaughter was in running list but was completed. Accepting new attempt. 2023-04-12 02:53:56,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40281] procedure.ProcedureCoordinator(165): Submitting procedure testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,427 INFO [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testSplitTableRegionUnevenDaughter' 2023-04-12 02:53:56,427 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:53:56,428 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testSplitTableRegionUnevenDaughter' starting 'acquire' 2023-04-12 02:53:56,428 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testSplitTableRegionUnevenDaughter', kicking off acquire phase on members. 2023-04-12 02:53:56,429 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,429 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,430 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:53:56,430 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:53:56,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:53:56,430 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:53:56,430 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:53:56,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:53:56,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:53:56,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:53:56,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:53:56,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:53:56,431 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:53:56,431 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 02:53:56,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,431 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,432 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,432 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 02:53:56,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 02:53:56,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,432 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 02:53:56,432 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,432 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,432 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionUnevenDaughter is still present, removing 2023-04-12 02:53:56,432 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,432 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionUnevenDaughter is still present, removing 2023-04-12 02:53:56,433 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionUnevenDaughter is still present, removing 2023-04-12 02:53:56,433 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionUnevenDaughter' with timeout 60000ms 2023-04-12 02:53:56,433 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,433 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,433 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionUnevenDaughter' with timeout 60000ms 2023-04-12 02:53:56,433 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionUnevenDaughter' with timeout 60000ms 2023-04-12 02:53:56,433 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:53:56,433 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:53:56,434 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionUnevenDaughter' starting 'acquire' stage 2023-04-12 02:53:56,434 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionUnevenDaughter' locally acquired 2023-04-12 02:53:56,434 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionUnevenDaughter' starting 'acquire' stage 2023-04-12 02:53:56,433 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:53:56,434 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34579,1681268031719' joining acquired barrier for procedure (testSplitTableRegionUnevenDaughter) in zk 2023-04-12 02:53:56,436 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionUnevenDaughter' starting 'acquire' stage 2023-04-12 02:53:56,436 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 02:53:56,436 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 02:53:56,436 DEBUG [rs(jenkins-hbase4.apache.org,44679,1681268031772)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testSplitTableRegionUnevenDaughter,,1681268034745.cc9012ec0b656baa24d04368fbe026c4. 2023-04-12 02:53:56,436 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionUnevenDaughter' locally acquired 2023-04-12 02:53:56,436 DEBUG [rs(jenkins-hbase4.apache.org,44679,1681268031772)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testSplitTableRegionUnevenDaughter,,1681268034745.cc9012ec0b656baa24d04368fbe026c4. started... 2023-04-12 02:53:56,436 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43091,1681268031804' joining acquired barrier for procedure (testSplitTableRegionUnevenDaughter) in zk 2023-04-12 02:53:56,436 INFO [rs(jenkins-hbase4.apache.org,44679,1681268031772)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing cc9012ec0b656baa24d04368fbe026c4 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-04-12 02:53:56,437 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,439 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,439 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,439 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionUnevenDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 02:53:56,440 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,440 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionUnevenDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 02:53:56,453 INFO [rs(jenkins-hbase4.apache.org,44679,1681268031772)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=58 (bloomFilter=true), to=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4/.tmp/cf1/a32265ffc17446ddb889acfdd1472f9c 2023-04-12 02:53:56,485 INFO [rs(jenkins-hbase4.apache.org,44679,1681268031772)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=58 (bloomFilter=true), to=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4/.tmp/cf2/3310af0091c24e11ae1d57719b30d33b 2023-04-12 02:53:56,493 DEBUG [rs(jenkins-hbase4.apache.org,44679,1681268031772)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4/.tmp/cf1/a32265ffc17446ddb889acfdd1472f9c as hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4/cf1/a32265ffc17446ddb889acfdd1472f9c 2023-04-12 02:53:56,499 INFO [rs(jenkins-hbase4.apache.org,44679,1681268031772)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4/cf1/a32265ffc17446ddb889acfdd1472f9c, entries=10, sequenceid=58, filesize=5.0 K 2023-04-12 02:53:56,500 DEBUG [rs(jenkins-hbase4.apache.org,44679,1681268031772)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4/.tmp/cf2/3310af0091c24e11ae1d57719b30d33b as hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4/cf2/3310af0091c24e11ae1d57719b30d33b 2023-04-12 02:53:56,508 INFO [rs(jenkins-hbase4.apache.org,44679,1681268031772)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4/cf2/3310af0091c24e11ae1d57719b30d33b, entries=10, sequenceid=58, filesize=5.0 K 2023-04-12 02:53:56,509 INFO [rs(jenkins-hbase4.apache.org,44679,1681268031772)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for cc9012ec0b656baa24d04368fbe026c4 in 73ms, sequenceid=58, compaction requested=true 2023-04-12 02:53:56,509 DEBUG [rs(jenkins-hbase4.apache.org,44679,1681268031772)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for cc9012ec0b656baa24d04368fbe026c4: 2023-04-12 02:53:56,509 DEBUG [rs(jenkins-hbase4.apache.org,44679,1681268031772)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testSplitTableRegionUnevenDaughter,,1681268034745.cc9012ec0b656baa24d04368fbe026c4. 2023-04-12 02:53:56,509 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 02:53:56,510 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 02:53:56,510 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:53:56,510 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionUnevenDaughter' locally acquired 2023-04-12 02:53:56,510 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44679,1681268031772' joining acquired barrier for procedure (testSplitTableRegionUnevenDaughter) in zk 2023-04-12 02:53:56,512 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,512 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:53:56,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:53:56,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 02:53:56,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 02:53:56,512 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,513 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionUnevenDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 02:53:56,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 02:53:56,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 02:53:56,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:53:56,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:53:56,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:53:56,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 02:53:56,515 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44679,1681268031772' joining acquired barrier for procedure 'testSplitTableRegionUnevenDaughter' on coordinator 2023-04-12 02:53:56,515 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testSplitTableRegionUnevenDaughter' starting 'in-barrier' execution. 2023-04-12 02:53:56,515 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6f6dc652[Count = 0] remaining members to acquire global barrier 2023-04-12 02:53:56,515 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,516 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,516 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,516 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,516 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,516 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,516 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionUnevenDaughter' received 'reached' from coordinator. 2023-04-12 02:53:56,516 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,516 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:53:56,516 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 02:53:56,516 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionUnevenDaughter' locally completed 2023-04-12 02:53:56,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,516 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionUnevenDaughter' completed for member 'jenkins-hbase4.apache.org,43091,1681268031804' in zk 2023-04-12 02:53:56,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,517 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionUnevenDaughter' received 'reached' from coordinator. 2023-04-12 02:53:56,517 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionUnevenDaughter' received 'reached' from coordinator. 2023-04-12 02:53:56,517 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionUnevenDaughter' locally completed 2023-04-12 02:53:56,517 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionUnevenDaughter' completed for member 'jenkins-hbase4.apache.org,44679,1681268031772' in zk 2023-04-12 02:53:56,517 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionUnevenDaughter' locally completed 2023-04-12 02:53:56,517 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionUnevenDaughter' completed for member 'jenkins-hbase4.apache.org,34579,1681268031719' in zk 2023-04-12 02:53:56,518 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionUnevenDaughter' has notified controller of completion 2023-04-12 02:53:56,518 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:53:56,518 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionUnevenDaughter' completed. 2023-04-12 02:53:56,520 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionUnevenDaughter' has notified controller of completion 2023-04-12 02:53:56,520 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionUnevenDaughter' has notified controller of completion 2023-04-12 02:53:56,520 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:53:56,520 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:53:56,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:53:56,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 02:53:56,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 02:53:56,520 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:53:56,520 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionUnevenDaughter' completed. 2023-04-12 02:53:56,521 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionUnevenDaughter' completed. 2023-04-12 02:53:56,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 02:53:56,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 02:53:56,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:53:56,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:53:56,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:53:56,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 02:53:56,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:53:56,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:53:56,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:53:56,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testSplitTableRegionUnevenDaughter' member 'jenkins-hbase4.apache.org,44679,1681268031772': 2023-04-12 02:53:56,524 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44679,1681268031772' released barrier for procedure'testSplitTableRegionUnevenDaughter', counting down latch. Waiting for 0 more 2023-04-12 02:53:56,524 INFO [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testSplitTableRegionUnevenDaughter' execution completed 2023-04-12 02:53:56,524 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 02:53:56,525 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 02:53:56,525 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,525 INFO [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testSplitTableRegionUnevenDaughterincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 02:53:56,527 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,527 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,527 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,527 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,527 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,527 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:53:56,527 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,527 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:53:56,527 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,527 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:53:56,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 02:53:56,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 02:53:56,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,527 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:53:56,527 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:53:56,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:53:56,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:53:56,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 02:53:56,528 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:53:56,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:53:56,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,528 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:53:56,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,529 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:53:56,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 02:53:56,529 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:53:56,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:53:56,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:53:56,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:53:56,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 02:53:56,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,531 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:53:56,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:53:56,531 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:53:56,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:53:56,532 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:53:56,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:53:56,538 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:53:56,538 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,538 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:53:56,538 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:53:56,538 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:53:56,538 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:53:56,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:53:56,538 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40281] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testSplitTableRegionUnevenDaughter' 2023-04-12 02:53:56,538 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40281] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 02:53:56,538 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:53:56,538 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:53:56,538 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:53:56,538 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:53:56,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:53:56,538 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:53:56,538 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:53:56,539 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:53:56,539 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:53:56,538 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,539 DEBUG [Listener at localhost/46667] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testSplitTableRegionUnevenDaughter'' to complete. (max 20000 ms per retry) 2023-04-12 02:53:56,538 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:53:56,539 DEBUG [Listener at localhost/46667] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 02:53:56,539 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:53:56,539 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:53:56,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:53:56,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:53:56,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:53:56,540 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,540 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:53:56,540 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:53:56,540 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,540 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,540 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:53:56,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:53:56,640 DEBUG [Listener at localhost/46667] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 02:53:56,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40281] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 02:53:56,665 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40281] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 02:53:56,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40281] procedure.ProcedureCoordinator(143): Procedure testSplitTableRegionUnevenDaughter was in running list but was completed. Accepting new attempt. 2023-04-12 02:53:56,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40281] procedure.ProcedureCoordinator(165): Submitting procedure testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,668 INFO [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testSplitTableRegionUnevenDaughter' 2023-04-12 02:53:56,668 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:53:56,669 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testSplitTableRegionUnevenDaughter' starting 'acquire' 2023-04-12 02:53:56,669 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testSplitTableRegionUnevenDaughter', kicking off acquire phase on members. 2023-04-12 02:53:56,669 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,669 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,671 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:53:56,671 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:53:56,671 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:53:56,671 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:53:56,671 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:53:56,671 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:53:56,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:53:56,671 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:53:56,671 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 02:53:56,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:53:56,671 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:53:56,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:53:56,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,672 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,672 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,672 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 02:53:56,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 02:53:56,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,672 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 02:53:56,672 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,673 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,673 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionUnevenDaughter is still present, removing 2023-04-12 02:53:56,673 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionUnevenDaughter is still present, removing 2023-04-12 02:53:56,673 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,673 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionUnevenDaughter is still present, removing 2023-04-12 02:53:56,673 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionUnevenDaughter' with timeout 60000ms 2023-04-12 02:53:56,673 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:53:56,673 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,674 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionUnevenDaughter' starting 'acquire' stage 2023-04-12 02:53:56,674 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionUnevenDaughter' with timeout 60000ms 2023-04-12 02:53:56,673 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,674 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:53:56,674 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionUnevenDaughter' with timeout 60000ms 2023-04-12 02:53:56,674 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionUnevenDaughter' starting 'acquire' stage 2023-04-12 02:53:56,674 DEBUG [rs(jenkins-hbase4.apache.org,44679,1681268031772)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testSplitTableRegionUnevenDaughter,,1681268034745.cc9012ec0b656baa24d04368fbe026c4. 2023-04-12 02:53:56,674 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:53:56,674 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 02:53:56,675 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 02:53:56,675 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionUnevenDaughter' starting 'acquire' stage 2023-04-12 02:53:56,675 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionUnevenDaughter' locally acquired 2023-04-12 02:53:56,674 DEBUG [rs(jenkins-hbase4.apache.org,44679,1681268031772)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testSplitTableRegionUnevenDaughter,,1681268034745.cc9012ec0b656baa24d04368fbe026c4. started... 2023-04-12 02:53:56,674 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionUnevenDaughter' locally acquired 2023-04-12 02:53:56,675 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43091,1681268031804' joining acquired barrier for procedure (testSplitTableRegionUnevenDaughter) in zk 2023-04-12 02:53:56,675 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34579,1681268031719' joining acquired barrier for procedure (testSplitTableRegionUnevenDaughter) in zk 2023-04-12 02:53:56,675 INFO [rs(jenkins-hbase4.apache.org,44679,1681268031772)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing cc9012ec0b656baa24d04368fbe026c4 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-04-12 02:53:56,678 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,680 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,680 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,680 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionUnevenDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 02:53:56,680 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,680 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionUnevenDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 02:53:56,693 INFO [rs(jenkins-hbase4.apache.org,44679,1681268031772)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=71 (bloomFilter=true), to=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4/.tmp/cf1/73d3880f14274287a94d30f14b9cf96e 2023-04-12 02:53:56,715 INFO [rs(jenkins-hbase4.apache.org,44679,1681268031772)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=71 (bloomFilter=true), to=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4/.tmp/cf2/260b76cda6d74251b52568d0821466a2 2023-04-12 02:53:56,724 DEBUG [rs(jenkins-hbase4.apache.org,44679,1681268031772)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4/.tmp/cf1/73d3880f14274287a94d30f14b9cf96e as hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4/cf1/73d3880f14274287a94d30f14b9cf96e 2023-04-12 02:53:56,735 INFO [rs(jenkins-hbase4.apache.org,44679,1681268031772)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4/cf1/73d3880f14274287a94d30f14b9cf96e, entries=10, sequenceid=71, filesize=5.0 K 2023-04-12 02:53:56,736 DEBUG [rs(jenkins-hbase4.apache.org,44679,1681268031772)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4/.tmp/cf2/260b76cda6d74251b52568d0821466a2 as hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4/cf2/260b76cda6d74251b52568d0821466a2 2023-04-12 02:53:56,746 INFO [rs(jenkins-hbase4.apache.org,44679,1681268031772)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4/cf2/260b76cda6d74251b52568d0821466a2, entries=10, sequenceid=71, filesize=5.0 K 2023-04-12 02:53:56,747 INFO [rs(jenkins-hbase4.apache.org,44679,1681268031772)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for cc9012ec0b656baa24d04368fbe026c4 in 72ms, sequenceid=71, compaction requested=true 2023-04-12 02:53:56,747 DEBUG [rs(jenkins-hbase4.apache.org,44679,1681268031772)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for cc9012ec0b656baa24d04368fbe026c4: 2023-04-12 02:53:56,747 DEBUG [rs(jenkins-hbase4.apache.org,44679,1681268031772)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testSplitTableRegionUnevenDaughter,,1681268034745.cc9012ec0b656baa24d04368fbe026c4. 2023-04-12 02:53:56,747 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 02:53:56,750 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 02:53:56,750 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:53:56,750 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionUnevenDaughter' locally acquired 2023-04-12 02:53:56,750 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44679,1681268031772' joining acquired barrier for procedure (testSplitTableRegionUnevenDaughter) in zk 2023-04-12 02:53:56,752 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,752 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:53:56,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:53:56,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 02:53:56,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 02:53:56,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 02:53:56,753 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,753 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionUnevenDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 02:53:56,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 02:53:56,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:53:56,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:53:56,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:53:56,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 02:53:56,757 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44679,1681268031772' joining acquired barrier for procedure 'testSplitTableRegionUnevenDaughter' on coordinator 2023-04-12 02:53:56,757 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@200345b[Count = 0] remaining members to acquire global barrier 2023-04-12 02:53:56,757 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testSplitTableRegionUnevenDaughter' starting 'in-barrier' execution. 2023-04-12 02:53:56,757 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,758 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,758 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,758 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,759 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,759 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:53:56,759 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionUnevenDaughter' received 'reached' from coordinator. 2023-04-12 02:53:56,759 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionUnevenDaughter' locally completed 2023-04-12 02:53:56,758 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,759 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionUnevenDaughter' completed for member 'jenkins-hbase4.apache.org,34579,1681268031719' in zk 2023-04-12 02:53:56,759 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,759 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 02:53:56,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,759 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionUnevenDaughter' received 'reached' from coordinator. 2023-04-12 02:53:56,759 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionUnevenDaughter' received 'reached' from coordinator. 2023-04-12 02:53:56,759 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionUnevenDaughter' locally completed 2023-04-12 02:53:56,759 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionUnevenDaughter' locally completed 2023-04-12 02:53:56,760 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionUnevenDaughter' completed for member 'jenkins-hbase4.apache.org,44679,1681268031772' in zk 2023-04-12 02:53:56,759 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionUnevenDaughter' completed for member 'jenkins-hbase4.apache.org,43091,1681268031804' in zk 2023-04-12 02:53:56,762 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionUnevenDaughter' has notified controller of completion 2023-04-12 02:53:56,762 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:53:56,762 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionUnevenDaughter' completed. 2023-04-12 02:53:56,764 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionUnevenDaughter' has notified controller of completion 2023-04-12 02:53:56,764 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:53:56,764 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionUnevenDaughter' completed. 2023-04-12 02:53:56,766 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:53:56,766 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionUnevenDaughter' has notified controller of completion 2023-04-12 02:53:56,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:53:56,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 02:53:56,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 02:53:56,766 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:53:56,766 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionUnevenDaughter' completed. 2023-04-12 02:53:56,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 02:53:56,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 02:53:56,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:53:56,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:53:56,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:53:56,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 02:53:56,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:53:56,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:53:56,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:53:56,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testSplitTableRegionUnevenDaughter' member 'jenkins-hbase4.apache.org,44679,1681268031772': 2023-04-12 02:53:56,779 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44679,1681268031772' released barrier for procedure'testSplitTableRegionUnevenDaughter', counting down latch. Waiting for 0 more 2023-04-12 02:53:56,779 INFO [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testSplitTableRegionUnevenDaughter' execution completed 2023-04-12 02:53:56,779 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 02:53:56,779 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 02:53:56,779 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,779 INFO [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testSplitTableRegionUnevenDaughterincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 02:53:56,786 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,786 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,786 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,786 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 02:53:56,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 02:53:56,786 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:53:56,786 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,786 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:53:56,786 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,786 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:53:56,787 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,787 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:53:56,787 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:53:56,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 02:53:56,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:53:56,800 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:53:56,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:53:56,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,800 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:53:56,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:53:56,800 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:53:56,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 02:53:56,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,801 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:53:56,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:53:56,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:53:56,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:53:56,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 02:53:56,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,803 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:53:56,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:53:56,804 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:53:56,804 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:53:56,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:53:56,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:53:56,808 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:53:56,808 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,808 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:53:56,808 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:53:56,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:53:56,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:53:56,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:53:56,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:53:56,808 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:53:56,808 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:53:56,808 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40281] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testSplitTableRegionUnevenDaughter' 2023-04-12 02:53:56,808 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:53:56,808 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:53:56,809 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:53:56,809 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:53:56,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:53:56,808 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:53:56,809 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:53:56,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:53:56,809 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:53:56,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:53:56,808 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40281] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 02:53:56,808 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:53:56,810 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,810 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:53:56,810 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,810 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:53:56,810 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:53:56,810 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:53:56,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:53:56,810 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,810 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-12 02:53:56,810 DEBUG [Listener at localhost/46667] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testSplitTableRegionUnevenDaughter'' to complete. (max 20000 ms per retry) 2023-04-12 02:53:56,811 DEBUG [Listener at localhost/46667] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 02:53:56,911 DEBUG [Listener at localhost/46667] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 02:53:56,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40281] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 02:53:56,948 DEBUG [Listener at localhost/46667] assignment.SplitTableRegionProcedure(219): Splittable=true state=OPEN, location=jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:53:56,958 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(1029): Stored pid=12, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=cc9012ec0b656baa24d04368fbe026c4, daughterA=e2a9b8fe15a781798c0b49e48ff5aad7, daughterB=6f55b5d6a460bd8c7569dee22f049104 2023-04-12 02:53:56,959 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=12, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=cc9012ec0b656baa24d04368fbe026c4, daughterA=e2a9b8fe15a781798c0b49e48ff5aad7, daughterB=6f55b5d6a460bd8c7569dee22f049104 2023-04-12 02:53:56,959 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=12, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=cc9012ec0b656baa24d04368fbe026c4, daughterA=e2a9b8fe15a781798c0b49e48ff5aad7, daughterB=6f55b5d6a460bd8c7569dee22f049104 2023-04-12 02:53:56,959 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=12, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=cc9012ec0b656baa24d04368fbe026c4, daughterA=e2a9b8fe15a781798c0b49e48ff5aad7, daughterB=6f55b5d6a460bd8c7569dee22f049104 2023-04-12 02:53:56,978 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitTableRegionUnevenDaughter, region=cc9012ec0b656baa24d04368fbe026c4, UNASSIGN}] 2023-04-12 02:53:56,980 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitTableRegionUnevenDaughter, region=cc9012ec0b656baa24d04368fbe026c4, UNASSIGN 2023-04-12 02:53:56,981 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=cc9012ec0b656baa24d04368fbe026c4, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:53:56,981 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitTableRegionUnevenDaughter,,1681268034745.cc9012ec0b656baa24d04368fbe026c4.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1681268036981"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681268036981"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681268036981"}]},"ts":"1681268036981"} 2023-04-12 02:53:56,985 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=14, ppid=13, state=RUNNABLE; CloseRegionProcedure cc9012ec0b656baa24d04368fbe026c4, server=jenkins-hbase4.apache.org,44679,1681268031772}] 2023-04-12 02:53:57,153 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close cc9012ec0b656baa24d04368fbe026c4 2023-04-12 02:53:57,153 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing cc9012ec0b656baa24d04368fbe026c4, disabling compactions & flushes 2023-04-12 02:53:57,153 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region testSplitTableRegionUnevenDaughter,,1681268034745.cc9012ec0b656baa24d04368fbe026c4. 2023-04-12 02:53:57,153 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testSplitTableRegionUnevenDaughter,,1681268034745.cc9012ec0b656baa24d04368fbe026c4. 2023-04-12 02:53:57,153 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on testSplitTableRegionUnevenDaughter,,1681268034745.cc9012ec0b656baa24d04368fbe026c4. after waiting 0 ms 2023-04-12 02:53:57,153 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region testSplitTableRegionUnevenDaughter,,1681268034745.cc9012ec0b656baa24d04368fbe026c4. 2023-04-12 02:53:57,154 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing cc9012ec0b656baa24d04368fbe026c4 2/2 column families, dataSize=480 B heapSize=2.13 KB 2023-04-12 02:53:57,249 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=240 B at sequenceid=82 (bloomFilter=true), to=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4/.tmp/cf1/5743ff156f3a424f80d4de48581aeeae 2023-04-12 02:53:57,371 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=240 B at sequenceid=82 (bloomFilter=true), to=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4/.tmp/cf2/7ed05e97ddbc4e359abe117e7e8b3de9 2023-04-12 02:53:57,391 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4/.tmp/cf1/5743ff156f3a424f80d4de48581aeeae as hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4/cf1/5743ff156f3a424f80d4de48581aeeae 2023-04-12 02:53:57,400 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4/cf1/5743ff156f3a424f80d4de48581aeeae, entries=8, sequenceid=82, filesize=5.0 K 2023-04-12 02:53:57,406 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4/.tmp/cf2/7ed05e97ddbc4e359abe117e7e8b3de9 as hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4/cf2/7ed05e97ddbc4e359abe117e7e8b3de9 2023-04-12 02:53:57,430 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4/cf2/7ed05e97ddbc4e359abe117e7e8b3de9, entries=8, sequenceid=82, filesize=5.0 K 2023-04-12 02:53:57,432 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~480 B/480, heapSize ~2.09 KB/2144, currentSize=0 B/0 for cc9012ec0b656baa24d04368fbe026c4 in 279ms, sequenceid=82, compaction requested=true 2023-04-12 02:53:57,522 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4/recovered.edits/85.seqid, newMaxSeqId=85, maxSeqId=1 2023-04-12 02:53:57,523 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro 2023-04-12 02:53:57,524 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed testSplitTableRegionUnevenDaughter,,1681268034745.cc9012ec0b656baa24d04368fbe026c4. 2023-04-12 02:53:57,524 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for cc9012ec0b656baa24d04368fbe026c4: 2023-04-12 02:53:57,527 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed cc9012ec0b656baa24d04368fbe026c4 2023-04-12 02:53:57,528 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=cc9012ec0b656baa24d04368fbe026c4, regionState=CLOSED 2023-04-12 02:53:57,529 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testSplitTableRegionUnevenDaughter,,1681268034745.cc9012ec0b656baa24d04368fbe026c4.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1681268037528"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681268037528"}]},"ts":"1681268037528"} 2023-04-12 02:53:57,534 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=14, resume processing ppid=13 2023-04-12 02:53:57,535 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=14, ppid=13, state=SUCCESS; CloseRegionProcedure cc9012ec0b656baa24d04368fbe026c4, server=jenkins-hbase4.apache.org,44679,1681268031772 in 546 msec 2023-04-12 02:53:57,538 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=13, resume processing ppid=12 2023-04-12 02:53:57,538 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=13, ppid=12, state=SUCCESS; TransitRegionStateProcedure table=testSplitTableRegionUnevenDaughter, region=cc9012ec0b656baa24d04368fbe026c4, UNASSIGN in 557 msec 2023-04-12 02:53:57,571 INFO [PEWorker-1] assignment.SplitTableRegionProcedure(694): pid=12 splitting 14 storefiles, region=cc9012ec0b656baa24d04368fbe026c4, threads=14 2023-04-12 02:53:57,573 DEBUG [StoreFileSplitter-pool-0] assignment.SplitTableRegionProcedure(776): pid=12 splitting started for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4/cf2/18c4c7a41a6d44f9a7eb1d9d77806e17 for region: cc9012ec0b656baa24d04368fbe026c4 2023-04-12 02:53:57,573 DEBUG [StoreFileSplitter-pool-1] assignment.SplitTableRegionProcedure(776): pid=12 splitting started for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4/cf2/1b7cac025fb940cab9783583b39cef46 for region: cc9012ec0b656baa24d04368fbe026c4 2023-04-12 02:53:57,574 DEBUG [StoreFileSplitter-pool-2] assignment.SplitTableRegionProcedure(776): pid=12 splitting started for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4/cf2/213df881d096427c9a5864569db2e1c4 for region: cc9012ec0b656baa24d04368fbe026c4 2023-04-12 02:53:57,577 DEBUG [StoreFileSplitter-pool-3] assignment.SplitTableRegionProcedure(776): pid=12 splitting started for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4/cf2/260b76cda6d74251b52568d0821466a2 for region: cc9012ec0b656baa24d04368fbe026c4 2023-04-12 02:53:57,577 DEBUG [StoreFileSplitter-pool-4] assignment.SplitTableRegionProcedure(776): pid=12 splitting started for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4/cf2/3310af0091c24e11ae1d57719b30d33b for region: cc9012ec0b656baa24d04368fbe026c4 2023-04-12 02:53:57,585 DEBUG [StoreFileSplitter-pool-5] assignment.SplitTableRegionProcedure(776): pid=12 splitting started for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4/cf2/7ed05e97ddbc4e359abe117e7e8b3de9 for region: cc9012ec0b656baa24d04368fbe026c4 2023-04-12 02:53:57,588 DEBUG [StoreFileSplitter-pool-6] assignment.SplitTableRegionProcedure(776): pid=12 splitting started for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4/cf2/8df43cbe10c74ad8b2c721f3509bb1d1 for region: cc9012ec0b656baa24d04368fbe026c4 2023-04-12 02:53:57,594 DEBUG [StoreFileSplitter-pool-7] assignment.SplitTableRegionProcedure(776): pid=12 splitting started for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4/cf1/07c3d7abf05f43cdbe44fe637da34a7d for region: cc9012ec0b656baa24d04368fbe026c4 2023-04-12 02:53:57,607 DEBUG [StoreFileSplitter-pool-8] assignment.SplitTableRegionProcedure(776): pid=12 splitting started for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4/cf1/364127255e644c3d8f0a55272e1ab83a for region: cc9012ec0b656baa24d04368fbe026c4 2023-04-12 02:53:57,609 DEBUG [StoreFileSplitter-pool-9] assignment.SplitTableRegionProcedure(776): pid=12 splitting started for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4/cf1/5743ff156f3a424f80d4de48581aeeae for region: cc9012ec0b656baa24d04368fbe026c4 2023-04-12 02:53:57,621 DEBUG [StoreFileSplitter-pool-10] assignment.SplitTableRegionProcedure(776): pid=12 splitting started for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4/cf1/73d3880f14274287a94d30f14b9cf96e for region: cc9012ec0b656baa24d04368fbe026c4 2023-04-12 02:53:57,634 DEBUG [StoreFileSplitter-pool-11] assignment.SplitTableRegionProcedure(776): pid=12 splitting started for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4/cf1/a32265ffc17446ddb889acfdd1472f9c for region: cc9012ec0b656baa24d04368fbe026c4 2023-04-12 02:53:57,641 DEBUG [StoreFileSplitter-pool-12] assignment.SplitTableRegionProcedure(776): pid=12 splitting started for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4/cf1/a859f22056324f46889984d0e730b4bf for region: cc9012ec0b656baa24d04368fbe026c4 2023-04-12 02:53:57,645 DEBUG [StoreFileSplitter-pool-13] assignment.SplitTableRegionProcedure(776): pid=12 splitting started for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4/cf1/b2cffa1e1e9f40ba81d73ca9452dfeb4 for region: cc9012ec0b656baa24d04368fbe026c4 2023-04-12 02:53:57,823 DEBUG [StoreFileSplitter-pool-3] regionserver.HRegionFileSystem(700): Will create HFileLink file for hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4/cf2/260b76cda6d74251b52568d0821466a2, top=true 2023-04-12 02:53:57,825 DEBUG [StoreFileSplitter-pool-11] regionserver.HRegionFileSystem(700): Will create HFileLink file for hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4/cf1/a32265ffc17446ddb889acfdd1472f9c, top=true 2023-04-12 02:53:57,832 DEBUG [StoreFileSplitter-pool-4] regionserver.HRegionFileSystem(700): Will create HFileLink file for hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4/cf2/3310af0091c24e11ae1d57719b30d33b, top=true 2023-04-12 02:53:57,867 DEBUG [StoreFileSplitter-pool-9] regionserver.HRegionFileSystem(700): Will create HFileLink file for hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4/cf1/5743ff156f3a424f80d4de48581aeeae, top=true 2023-04-12 02:53:57,921 DEBUG [StoreFileSplitter-pool-10] regionserver.HRegionFileSystem(700): Will create HFileLink file for hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4/cf1/73d3880f14274287a94d30f14b9cf96e, top=true 2023-04-12 02:53:57,921 DEBUG [StoreFileSplitter-pool-5] regionserver.HRegionFileSystem(700): Will create HFileLink file for hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4/cf2/7ed05e97ddbc4e359abe117e7e8b3de9, top=true 2023-04-12 02:53:57,996 INFO [StoreFileSplitter-pool-11] regionserver.HRegionFileSystem(742): Created linkFile:hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4/.splits/6f55b5d6a460bd8c7569dee22f049104/cf1/testSplitTableRegionUnevenDaughter=cc9012ec0b656baa24d04368fbe026c4-a32265ffc17446ddb889acfdd1472f9c for child: 6f55b5d6a460bd8c7569dee22f049104, parent: cc9012ec0b656baa24d04368fbe026c4 2023-04-12 02:53:57,996 DEBUG [StoreFileSplitter-pool-11] assignment.SplitTableRegionProcedure(787): pid=12 splitting complete for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4/cf1/a32265ffc17446ddb889acfdd1472f9c for region: cc9012ec0b656baa24d04368fbe026c4 2023-04-12 02:53:58,035 INFO [StoreFileSplitter-pool-5] regionserver.HRegionFileSystem(742): Created linkFile:hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4/.splits/6f55b5d6a460bd8c7569dee22f049104/cf2/testSplitTableRegionUnevenDaughter=cc9012ec0b656baa24d04368fbe026c4-7ed05e97ddbc4e359abe117e7e8b3de9 for child: 6f55b5d6a460bd8c7569dee22f049104, parent: cc9012ec0b656baa24d04368fbe026c4 2023-04-12 02:53:58,036 INFO [StoreFileSplitter-pool-9] regionserver.HRegionFileSystem(742): Created linkFile:hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4/.splits/6f55b5d6a460bd8c7569dee22f049104/cf1/testSplitTableRegionUnevenDaughter=cc9012ec0b656baa24d04368fbe026c4-5743ff156f3a424f80d4de48581aeeae for child: 6f55b5d6a460bd8c7569dee22f049104, parent: cc9012ec0b656baa24d04368fbe026c4 2023-04-12 02:53:58,036 DEBUG [StoreFileSplitter-pool-9] assignment.SplitTableRegionProcedure(787): pid=12 splitting complete for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4/cf1/5743ff156f3a424f80d4de48581aeeae for region: cc9012ec0b656baa24d04368fbe026c4 2023-04-12 02:53:58,036 DEBUG [StoreFileSplitter-pool-5] assignment.SplitTableRegionProcedure(787): pid=12 splitting complete for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4/cf2/7ed05e97ddbc4e359abe117e7e8b3de9 for region: cc9012ec0b656baa24d04368fbe026c4 2023-04-12 02:53:58,036 INFO [StoreFileSplitter-pool-4] regionserver.HRegionFileSystem(742): Created linkFile:hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4/.splits/6f55b5d6a460bd8c7569dee22f049104/cf2/testSplitTableRegionUnevenDaughter=cc9012ec0b656baa24d04368fbe026c4-3310af0091c24e11ae1d57719b30d33b for child: 6f55b5d6a460bd8c7569dee22f049104, parent: cc9012ec0b656baa24d04368fbe026c4 2023-04-12 02:53:58,036 DEBUG [StoreFileSplitter-pool-4] assignment.SplitTableRegionProcedure(787): pid=12 splitting complete for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4/cf2/3310af0091c24e11ae1d57719b30d33b for region: cc9012ec0b656baa24d04368fbe026c4 2023-04-12 02:53:58,040 INFO [StoreFileSplitter-pool-3] regionserver.HRegionFileSystem(742): Created linkFile:hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4/.splits/6f55b5d6a460bd8c7569dee22f049104/cf2/testSplitTableRegionUnevenDaughter=cc9012ec0b656baa24d04368fbe026c4-260b76cda6d74251b52568d0821466a2 for child: 6f55b5d6a460bd8c7569dee22f049104, parent: cc9012ec0b656baa24d04368fbe026c4 2023-04-12 02:53:58,040 DEBUG [StoreFileSplitter-pool-3] assignment.SplitTableRegionProcedure(787): pid=12 splitting complete for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4/cf2/260b76cda6d74251b52568d0821466a2 for region: cc9012ec0b656baa24d04368fbe026c4 2023-04-12 02:53:58,090 DEBUG [StoreFileSplitter-pool-0] assignment.SplitTableRegionProcedure(787): pid=12 splitting complete for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4/cf2/18c4c7a41a6d44f9a7eb1d9d77806e17 for region: cc9012ec0b656baa24d04368fbe026c4 2023-04-12 02:53:58,116 INFO [StoreFileSplitter-pool-10] regionserver.HRegionFileSystem(742): Created linkFile:hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4/.splits/6f55b5d6a460bd8c7569dee22f049104/cf1/testSplitTableRegionUnevenDaughter=cc9012ec0b656baa24d04368fbe026c4-73d3880f14274287a94d30f14b9cf96e for child: 6f55b5d6a460bd8c7569dee22f049104, parent: cc9012ec0b656baa24d04368fbe026c4 2023-04-12 02:53:58,116 DEBUG [StoreFileSplitter-pool-10] assignment.SplitTableRegionProcedure(787): pid=12 splitting complete for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4/cf1/73d3880f14274287a94d30f14b9cf96e for region: cc9012ec0b656baa24d04368fbe026c4 2023-04-12 02:53:58,212 DEBUG [StoreFileSplitter-pool-7] assignment.SplitTableRegionProcedure(787): pid=12 splitting complete for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4/cf1/07c3d7abf05f43cdbe44fe637da34a7d for region: cc9012ec0b656baa24d04368fbe026c4 2023-04-12 02:53:58,248 DEBUG [StoreFileSplitter-pool-2] assignment.SplitTableRegionProcedure(787): pid=12 splitting complete for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4/cf2/213df881d096427c9a5864569db2e1c4 for region: cc9012ec0b656baa24d04368fbe026c4 2023-04-12 02:53:58,306 DEBUG [StoreFileSplitter-pool-8] assignment.SplitTableRegionProcedure(787): pid=12 splitting complete for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4/cf1/364127255e644c3d8f0a55272e1ab83a for region: cc9012ec0b656baa24d04368fbe026c4 2023-04-12 02:53:58,314 DEBUG [StoreFileSplitter-pool-1] assignment.SplitTableRegionProcedure(787): pid=12 splitting complete for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4/cf2/1b7cac025fb940cab9783583b39cef46 for region: cc9012ec0b656baa24d04368fbe026c4 2023-04-12 02:53:58,317 DEBUG [StoreFileSplitter-pool-13] assignment.SplitTableRegionProcedure(787): pid=12 splitting complete for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4/cf1/b2cffa1e1e9f40ba81d73ca9452dfeb4 for region: cc9012ec0b656baa24d04368fbe026c4 2023-04-12 02:53:58,521 DEBUG [StoreFileSplitter-pool-12] assignment.SplitTableRegionProcedure(787): pid=12 splitting complete for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4/cf1/a859f22056324f46889984d0e730b4bf for region: cc9012ec0b656baa24d04368fbe026c4 2023-04-12 02:53:58,559 DEBUG [StoreFileSplitter-pool-6] assignment.SplitTableRegionProcedure(787): pid=12 splitting complete for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4/cf2/8df43cbe10c74ad8b2c721f3509bb1d1 for region: cc9012ec0b656baa24d04368fbe026c4 2023-04-12 02:53:58,559 DEBUG [PEWorker-1] assignment.SplitTableRegionProcedure(755): pid=12 split storefiles for region cc9012ec0b656baa24d04368fbe026c4 Daughter A: 8 storefiles, Daughter B: 14 storefiles. 2023-04-12 02:53:58,746 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/e2a9b8fe15a781798c0b49e48ff5aad7/recovered.edits/85.seqid, newMaxSeqId=85, maxSeqId=-1 2023-04-12 02:53:58,755 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/6f55b5d6a460bd8c7569dee22f049104/recovered.edits/85.seqid, newMaxSeqId=85, maxSeqId=-1 2023-04-12 02:53:58,759 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":3,"row":"testSplitTableRegionUnevenDaughter,,1681268034745.cc9012ec0b656baa24d04368fbe026c4.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1681268038759"},{"qualifier":"splitA","vlen":70,"tag":[],"timestamp":"1681268038759"},{"qualifier":"splitB","vlen":70,"tag":[],"timestamp":"1681268038759"}]},"ts":"1681268038759"} 2023-04-12 02:53:58,760 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":3,"row":"testSplitTableRegionUnevenDaughter,,1681268036948.e2a9b8fe15a781798c0b49e48ff5aad7.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1681268038759"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681268038759"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681268038759"}]},"ts":"1681268038759"} 2023-04-12 02:53:58,760 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":3,"row":"testSplitTableRegionUnevenDaughter,26,1681268036948.6f55b5d6a460bd8c7569dee22f049104.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1681268038759"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681268038759"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681268038759"}]},"ts":"1681268038759"} 2023-04-12 02:53:58,843 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=15, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSplitTableRegionUnevenDaughter, region=e2a9b8fe15a781798c0b49e48ff5aad7, ASSIGN}, {pid=16, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSplitTableRegionUnevenDaughter, region=6f55b5d6a460bd8c7569dee22f049104, ASSIGN}] 2023-04-12 02:53:58,847 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=16, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSplitTableRegionUnevenDaughter, region=6f55b5d6a460bd8c7569dee22f049104, ASSIGN 2023-04-12 02:53:58,848 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=16, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testSplitTableRegionUnevenDaughter, region=6f55b5d6a460bd8c7569dee22f049104, ASSIGN; state=SPLITTING_NEW, location=jenkins-hbase4.apache.org,44679,1681268031772; forceNewPlan=false, retain=false 2023-04-12 02:53:58,851 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=15, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSplitTableRegionUnevenDaughter, region=e2a9b8fe15a781798c0b49e48ff5aad7, ASSIGN 2023-04-12 02:53:58,852 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=15, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testSplitTableRegionUnevenDaughter, region=e2a9b8fe15a781798c0b49e48ff5aad7, ASSIGN; state=SPLITTING_NEW, location=jenkins-hbase4.apache.org,44679,1681268031772; forceNewPlan=false, retain=false 2023-04-12 02:53:58,999 INFO [jenkins-hbase4:40281] balancer.BaseLoadBalancer(1545): Reassigned 2 regions. 2 retained the pre-restart assignment. 2023-04-12 02:53:59,001 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=15 updating hbase:meta row=e2a9b8fe15a781798c0b49e48ff5aad7, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:53:59,001 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitTableRegionUnevenDaughter,,1681268036948.e2a9b8fe15a781798c0b49e48ff5aad7.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1681268039001"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681268039001"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681268039001"}]},"ts":"1681268039001"} 2023-04-12 02:53:59,005 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=17, ppid=15, state=RUNNABLE; OpenRegionProcedure e2a9b8fe15a781798c0b49e48ff5aad7, server=jenkins-hbase4.apache.org,44679,1681268031772}] 2023-04-12 02:53:59,014 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=16 updating hbase:meta row=6f55b5d6a460bd8c7569dee22f049104, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:53:59,014 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitTableRegionUnevenDaughter,26,1681268036948.6f55b5d6a460bd8c7569dee22f049104.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1681268039014"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681268039014"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681268039014"}]},"ts":"1681268039014"} 2023-04-12 02:53:59,019 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=18, ppid=16, state=RUNNABLE; OpenRegionProcedure 6f55b5d6a460bd8c7569dee22f049104, server=jenkins-hbase4.apache.org,44679,1681268031772}] 2023-04-12 02:53:59,173 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testSplitTableRegionUnevenDaughter,26,1681268036948.6f55b5d6a460bd8c7569dee22f049104. 2023-04-12 02:53:59,173 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 6f55b5d6a460bd8c7569dee22f049104, NAME => 'testSplitTableRegionUnevenDaughter,26,1681268036948.6f55b5d6a460bd8c7569dee22f049104.', STARTKEY => '26', ENDKEY => ''} 2023-04-12 02:53:59,174 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(174): System coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro loaded, priority=536870911. 2023-04-12 02:53:59,174 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testSplitTableRegionUnevenDaughter 6f55b5d6a460bd8c7569dee22f049104 2023-04-12 02:53:59,174 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated testSplitTableRegionUnevenDaughter,26,1681268036948.6f55b5d6a460bd8c7569dee22f049104.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-12 02:53:59,174 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 6f55b5d6a460bd8c7569dee22f049104 2023-04-12 02:53:59,174 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 6f55b5d6a460bd8c7569dee22f049104 2023-04-12 02:53:59,197 INFO [StoreOpener-6f55b5d6a460bd8c7569dee22f049104-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf1 of region 6f55b5d6a460bd8c7569dee22f049104 2023-04-12 02:53:59,199 DEBUG [StoreOpener-6f55b5d6a460bd8c7569dee22f049104-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/6f55b5d6a460bd8c7569dee22f049104/cf1 2023-04-12 02:53:59,199 DEBUG [StoreOpener-6f55b5d6a460bd8c7569dee22f049104-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/6f55b5d6a460bd8c7569dee22f049104/cf1 2023-04-12 02:53:59,199 INFO [StoreOpener-6f55b5d6a460bd8c7569dee22f049104-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 6f55b5d6a460bd8c7569dee22f049104 columnFamilyName cf1 2023-04-12 02:53:59,223 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-12 02:53:59,291 DEBUG [StoreOpener-6f55b5d6a460bd8c7569dee22f049104-1] regionserver.HStore(539): loaded hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/6f55b5d6a460bd8c7569dee22f049104/cf1/07c3d7abf05f43cdbe44fe637da34a7d.cc9012ec0b656baa24d04368fbe026c4->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4/cf1/07c3d7abf05f43cdbe44fe637da34a7d-top 2023-04-12 02:53:59,314 DEBUG [StoreOpener-6f55b5d6a460bd8c7569dee22f049104-1] regionserver.HStore(539): loaded hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/6f55b5d6a460bd8c7569dee22f049104/cf1/364127255e644c3d8f0a55272e1ab83a.cc9012ec0b656baa24d04368fbe026c4->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4/cf1/364127255e644c3d8f0a55272e1ab83a-top 2023-04-12 02:53:59,342 DEBUG [StoreOpener-6f55b5d6a460bd8c7569dee22f049104-1] regionserver.HStore(539): loaded hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/6f55b5d6a460bd8c7569dee22f049104/cf1/a859f22056324f46889984d0e730b4bf.cc9012ec0b656baa24d04368fbe026c4->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4/cf1/a859f22056324f46889984d0e730b4bf-top 2023-04-12 02:53:59,367 DEBUG [StoreOpener-6f55b5d6a460bd8c7569dee22f049104-1] regionserver.HStore(539): loaded hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/6f55b5d6a460bd8c7569dee22f049104/cf1/b2cffa1e1e9f40ba81d73ca9452dfeb4.cc9012ec0b656baa24d04368fbe026c4->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4/cf1/b2cffa1e1e9f40ba81d73ca9452dfeb4-top 2023-04-12 02:53:59,383 DEBUG [StoreOpener-6f55b5d6a460bd8c7569dee22f049104-1] regionserver.HStore(539): loaded hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/6f55b5d6a460bd8c7569dee22f049104/cf1/testSplitTableRegionUnevenDaughter=cc9012ec0b656baa24d04368fbe026c4-5743ff156f3a424f80d4de48581aeeae 2023-04-12 02:53:59,402 DEBUG [StoreOpener-6f55b5d6a460bd8c7569dee22f049104-1] regionserver.HStore(539): loaded hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/6f55b5d6a460bd8c7569dee22f049104/cf1/testSplitTableRegionUnevenDaughter=cc9012ec0b656baa24d04368fbe026c4-73d3880f14274287a94d30f14b9cf96e 2023-04-12 02:53:59,403 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:meta' 2023-04-12 02:53:59,418 DEBUG [StoreOpener-6f55b5d6a460bd8c7569dee22f049104-1] regionserver.HStore(539): loaded hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/6f55b5d6a460bd8c7569dee22f049104/cf1/testSplitTableRegionUnevenDaughter=cc9012ec0b656baa24d04368fbe026c4-a32265ffc17446ddb889acfdd1472f9c 2023-04-12 02:53:59,422 INFO [StoreOpener-6f55b5d6a460bd8c7569dee22f049104-1] regionserver.HStore(310): Store=6f55b5d6a460bd8c7569dee22f049104/cf1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-12 02:53:59,423 INFO [StoreOpener-6f55b5d6a460bd8c7569dee22f049104-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf2 of region 6f55b5d6a460bd8c7569dee22f049104 2023-04-12 02:53:59,423 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:namespace' 2023-04-12 02:53:59,428 DEBUG [StoreOpener-6f55b5d6a460bd8c7569dee22f049104-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/6f55b5d6a460bd8c7569dee22f049104/cf2 2023-04-12 02:53:59,428 DEBUG [StoreOpener-6f55b5d6a460bd8c7569dee22f049104-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/6f55b5d6a460bd8c7569dee22f049104/cf2 2023-04-12 02:53:59,429 INFO [StoreOpener-6f55b5d6a460bd8c7569dee22f049104-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 6f55b5d6a460bd8c7569dee22f049104 columnFamilyName cf2 2023-04-12 02:53:59,454 DEBUG [StoreOpener-6f55b5d6a460bd8c7569dee22f049104-1] regionserver.HStore(539): loaded hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/6f55b5d6a460bd8c7569dee22f049104/cf2/18c4c7a41a6d44f9a7eb1d9d77806e17.cc9012ec0b656baa24d04368fbe026c4->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4/cf2/18c4c7a41a6d44f9a7eb1d9d77806e17-top 2023-04-12 02:53:59,463 DEBUG [StoreOpener-6f55b5d6a460bd8c7569dee22f049104-1] regionserver.HStore(539): loaded hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/6f55b5d6a460bd8c7569dee22f049104/cf2/1b7cac025fb940cab9783583b39cef46.cc9012ec0b656baa24d04368fbe026c4->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4/cf2/1b7cac025fb940cab9783583b39cef46-top 2023-04-12 02:53:59,473 DEBUG [StoreOpener-6f55b5d6a460bd8c7569dee22f049104-1] regionserver.HStore(539): loaded hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/6f55b5d6a460bd8c7569dee22f049104/cf2/213df881d096427c9a5864569db2e1c4.cc9012ec0b656baa24d04368fbe026c4->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4/cf2/213df881d096427c9a5864569db2e1c4-top 2023-04-12 02:53:59,481 DEBUG [StoreOpener-6f55b5d6a460bd8c7569dee22f049104-1] regionserver.HStore(539): loaded hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/6f55b5d6a460bd8c7569dee22f049104/cf2/8df43cbe10c74ad8b2c721f3509bb1d1.cc9012ec0b656baa24d04368fbe026c4->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4/cf2/8df43cbe10c74ad8b2c721f3509bb1d1-top 2023-04-12 02:53:59,489 DEBUG [StoreOpener-6f55b5d6a460bd8c7569dee22f049104-1] regionserver.HStore(539): loaded hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/6f55b5d6a460bd8c7569dee22f049104/cf2/testSplitTableRegionUnevenDaughter=cc9012ec0b656baa24d04368fbe026c4-260b76cda6d74251b52568d0821466a2 2023-04-12 02:53:59,499 DEBUG [StoreOpener-6f55b5d6a460bd8c7569dee22f049104-1] regionserver.HStore(539): loaded hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/6f55b5d6a460bd8c7569dee22f049104/cf2/testSplitTableRegionUnevenDaughter=cc9012ec0b656baa24d04368fbe026c4-3310af0091c24e11ae1d57719b30d33b 2023-04-12 02:53:59,506 DEBUG [StoreOpener-6f55b5d6a460bd8c7569dee22f049104-1] regionserver.HStore(539): loaded hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/6f55b5d6a460bd8c7569dee22f049104/cf2/testSplitTableRegionUnevenDaughter=cc9012ec0b656baa24d04368fbe026c4-7ed05e97ddbc4e359abe117e7e8b3de9 2023-04-12 02:53:59,507 INFO [StoreOpener-6f55b5d6a460bd8c7569dee22f049104-1] regionserver.HStore(310): Store=6f55b5d6a460bd8c7569dee22f049104/cf2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-12 02:53:59,511 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/6f55b5d6a460bd8c7569dee22f049104 2023-04-12 02:53:59,513 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/6f55b5d6a460bd8c7569dee22f049104 2023-04-12 02:53:59,517 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testSplitTableRegionUnevenDaughter descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-04-12 02:53:59,520 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 6f55b5d6a460bd8c7569dee22f049104 2023-04-12 02:53:59,522 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 6f55b5d6a460bd8c7569dee22f049104; next sequenceid=86; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=74178481, jitterRate=0.10534550249576569}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-04-12 02:53:59,522 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 6f55b5d6a460bd8c7569dee22f049104: 2023-04-12 02:53:59,523 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testSplitTableRegionUnevenDaughter,26,1681268036948.6f55b5d6a460bd8c7569dee22f049104., pid=18, masterSystemTime=1681268039161 2023-04-12 02:53:59,527 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: Opening Region; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 02:53:59,527 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: Opening Region; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 02:53:59,528 DEBUG [RS:1;jenkins-hbase4:44679-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-12 02:53:59,528 DEBUG [RS:1;jenkins-hbase4:44679-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-12 02:53:59,531 INFO [RS:1;jenkins-hbase4:44679-longCompactions-0] regionserver.HStore(1898): Keeping/Overriding Compaction request priority to -2147482648 for CF cf2 since it belongs to recently split daughter region testSplitTableRegionUnevenDaughter,26,1681268036948.6f55b5d6a460bd8c7569dee22f049104. 2023-04-12 02:53:59,531 INFO [RS:1;jenkins-hbase4:44679-shortCompactions-0] regionserver.HStore(1898): Keeping/Overriding Compaction request priority to -2147482648 for CF cf1 since it belongs to recently split daughter region testSplitTableRegionUnevenDaughter,26,1681268036948.6f55b5d6a460bd8c7569dee22f049104. 2023-04-12 02:53:59,531 DEBUG [RS:1;jenkins-hbase4:44679-longCompactions-0] regionserver.HStore(1912): 6f55b5d6a460bd8c7569dee22f049104/cf2 is initiating minor compaction (all files) 2023-04-12 02:53:59,531 DEBUG [RS:1;jenkins-hbase4:44679-shortCompactions-0] regionserver.HStore(1912): 6f55b5d6a460bd8c7569dee22f049104/cf1 is initiating minor compaction (all files) 2023-04-12 02:53:59,531 INFO [RS:1;jenkins-hbase4:44679-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 6f55b5d6a460bd8c7569dee22f049104/cf2 in testSplitTableRegionUnevenDaughter,26,1681268036948.6f55b5d6a460bd8c7569dee22f049104. 2023-04-12 02:53:59,531 INFO [RS:1;jenkins-hbase4:44679-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 6f55b5d6a460bd8c7569dee22f049104/cf1 in testSplitTableRegionUnevenDaughter,26,1681268036948.6f55b5d6a460bd8c7569dee22f049104. 2023-04-12 02:53:59,532 INFO [RS:1;jenkins-hbase4:44679-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/6f55b5d6a460bd8c7569dee22f049104/cf1/b2cffa1e1e9f40ba81d73ca9452dfeb4.cc9012ec0b656baa24d04368fbe026c4->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4/cf1/b2cffa1e1e9f40ba81d73ca9452dfeb4-top, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/6f55b5d6a460bd8c7569dee22f049104/cf1/07c3d7abf05f43cdbe44fe637da34a7d.cc9012ec0b656baa24d04368fbe026c4->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4/cf1/07c3d7abf05f43cdbe44fe637da34a7d-top, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/6f55b5d6a460bd8c7569dee22f049104/cf1/364127255e644c3d8f0a55272e1ab83a.cc9012ec0b656baa24d04368fbe026c4->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4/cf1/364127255e644c3d8f0a55272e1ab83a-top, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/6f55b5d6a460bd8c7569dee22f049104/cf1/a859f22056324f46889984d0e730b4bf.cc9012ec0b656baa24d04368fbe026c4->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4/cf1/a859f22056324f46889984d0e730b4bf-top, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/6f55b5d6a460bd8c7569dee22f049104/cf1/testSplitTableRegionUnevenDaughter=cc9012ec0b656baa24d04368fbe026c4-a32265ffc17446ddb889acfdd1472f9c, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/6f55b5d6a460bd8c7569dee22f049104/cf1/testSplitTableRegionUnevenDaughter=cc9012ec0b656baa24d04368fbe026c4-73d3880f14274287a94d30f14b9cf96e, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/6f55b5d6a460bd8c7569dee22f049104/cf1/testSplitTableRegionUnevenDaughter=cc9012ec0b656baa24d04368fbe026c4-5743ff156f3a424f80d4de48581aeeae] into tmpdir=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/6f55b5d6a460bd8c7569dee22f049104/.tmp, totalSize=34.9 K 2023-04-12 02:53:59,532 INFO [RS:1;jenkins-hbase4:44679-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/6f55b5d6a460bd8c7569dee22f049104/cf2/18c4c7a41a6d44f9a7eb1d9d77806e17.cc9012ec0b656baa24d04368fbe026c4->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4/cf2/18c4c7a41a6d44f9a7eb1d9d77806e17-top, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/6f55b5d6a460bd8c7569dee22f049104/cf2/8df43cbe10c74ad8b2c721f3509bb1d1.cc9012ec0b656baa24d04368fbe026c4->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4/cf2/8df43cbe10c74ad8b2c721f3509bb1d1-top, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/6f55b5d6a460bd8c7569dee22f049104/cf2/1b7cac025fb940cab9783583b39cef46.cc9012ec0b656baa24d04368fbe026c4->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4/cf2/1b7cac025fb940cab9783583b39cef46-top, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/6f55b5d6a460bd8c7569dee22f049104/cf2/213df881d096427c9a5864569db2e1c4.cc9012ec0b656baa24d04368fbe026c4->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4/cf2/213df881d096427c9a5864569db2e1c4-top, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/6f55b5d6a460bd8c7569dee22f049104/cf2/testSplitTableRegionUnevenDaughter=cc9012ec0b656baa24d04368fbe026c4-3310af0091c24e11ae1d57719b30d33b, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/6f55b5d6a460bd8c7569dee22f049104/cf2/testSplitTableRegionUnevenDaughter=cc9012ec0b656baa24d04368fbe026c4-260b76cda6d74251b52568d0821466a2, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/6f55b5d6a460bd8c7569dee22f049104/cf2/testSplitTableRegionUnevenDaughter=cc9012ec0b656baa24d04368fbe026c4-7ed05e97ddbc4e359abe117e7e8b3de9] into tmpdir=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/6f55b5d6a460bd8c7569dee22f049104/.tmp, totalSize=34.9 K 2023-04-12 02:53:59,533 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testSplitTableRegionUnevenDaughter,26,1681268036948.6f55b5d6a460bd8c7569dee22f049104. 2023-04-12 02:53:59,534 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testSplitTableRegionUnevenDaughter,26,1681268036948.6f55b5d6a460bd8c7569dee22f049104. 2023-04-12 02:53:59,534 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testSplitTableRegionUnevenDaughter,,1681268036948.e2a9b8fe15a781798c0b49e48ff5aad7. 2023-04-12 02:53:59,535 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=16 updating hbase:meta row=6f55b5d6a460bd8c7569dee22f049104, regionState=OPEN, openSeqNum=86, regionLocation=jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:53:59,537 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => e2a9b8fe15a781798c0b49e48ff5aad7, NAME => 'testSplitTableRegionUnevenDaughter,,1681268036948.e2a9b8fe15a781798c0b49e48ff5aad7.', STARTKEY => '', ENDKEY => '26'} 2023-04-12 02:53:59,537 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testSplitTableRegionUnevenDaughter,26,1681268036948.6f55b5d6a460bd8c7569dee22f049104.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1681268039535"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1681268039535"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681268039535"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681268039535"}]},"ts":"1681268039535"} 2023-04-12 02:53:59,537 DEBUG [RS:1;jenkins-hbase4:44679-shortCompactions-0] compactions.Compactor(207): Compacting b2cffa1e1e9f40ba81d73ca9452dfeb4.cc9012ec0b656baa24d04368fbe026c4, keycount=1, bloomtype=ROW, size=4.8 K, encoding=NONE, compression=NONE, seqNum=7, earliestPutTs=1681268035253 2023-04-12 02:53:59,537 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(174): System coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro loaded, priority=536870911. 2023-04-12 02:53:59,538 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testSplitTableRegionUnevenDaughter e2a9b8fe15a781798c0b49e48ff5aad7 2023-04-12 02:53:59,538 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated testSplitTableRegionUnevenDaughter,,1681268036948.e2a9b8fe15a781798c0b49e48ff5aad7.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-12 02:53:59,538 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for e2a9b8fe15a781798c0b49e48ff5aad7 2023-04-12 02:53:59,538 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for e2a9b8fe15a781798c0b49e48ff5aad7 2023-04-12 02:53:59,552 DEBUG [RS:1;jenkins-hbase4:44679-longCompactions-0] compactions.Compactor(207): Compacting 18c4c7a41a6d44f9a7eb1d9d77806e17.cc9012ec0b656baa24d04368fbe026c4, keycount=1, bloomtype=ROW, size=4.8 K, encoding=NONE, compression=NONE, seqNum=7, earliestPutTs=1681268035253 2023-04-12 02:53:59,553 DEBUG [RS:1;jenkins-hbase4:44679-shortCompactions-0] compactions.Compactor(207): Compacting 07c3d7abf05f43cdbe44fe637da34a7d.cc9012ec0b656baa24d04368fbe026c4, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=20, earliestPutTs=1681268035652 2023-04-12 02:53:59,553 DEBUG [RS:1;jenkins-hbase4:44679-longCompactions-0] compactions.Compactor(207): Compacting 8df43cbe10c74ad8b2c721f3509bb1d1.cc9012ec0b656baa24d04368fbe026c4, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=20, earliestPutTs=1681268035652 2023-04-12 02:53:59,555 DEBUG [RS:1;jenkins-hbase4:44679-shortCompactions-0] compactions.Compactor(207): Compacting 364127255e644c3d8f0a55272e1ab83a.cc9012ec0b656baa24d04368fbe026c4, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=33, earliestPutTs=1681268035903 2023-04-12 02:53:59,555 DEBUG [RS:1;jenkins-hbase4:44679-longCompactions-0] compactions.Compactor(207): Compacting 1b7cac025fb940cab9783583b39cef46.cc9012ec0b656baa24d04368fbe026c4, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=33, earliestPutTs=1681268035903 2023-04-12 02:53:59,555 DEBUG [RS:1;jenkins-hbase4:44679-shortCompactions-0] compactions.Compactor(207): Compacting a859f22056324f46889984d0e730b4bf.cc9012ec0b656baa24d04368fbe026c4, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=46, earliestPutTs=1681268036147 2023-04-12 02:53:59,556 DEBUG [RS:1;jenkins-hbase4:44679-longCompactions-0] compactions.Compactor(207): Compacting 213df881d096427c9a5864569db2e1c4.cc9012ec0b656baa24d04368fbe026c4, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=46, earliestPutTs=1681268036147 2023-04-12 02:53:59,557 DEBUG [RS:1;jenkins-hbase4:44679-shortCompactions-0] compactions.Compactor(207): Compacting testSplitTableRegionUnevenDaughter=cc9012ec0b656baa24d04368fbe026c4-a32265ffc17446ddb889acfdd1472f9c, keycount=10, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=58, earliestPutTs=1681268036403 2023-04-12 02:53:59,557 DEBUG [RS:1;jenkins-hbase4:44679-longCompactions-0] compactions.Compactor(207): Compacting testSplitTableRegionUnevenDaughter=cc9012ec0b656baa24d04368fbe026c4-3310af0091c24e11ae1d57719b30d33b, keycount=10, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=58, earliestPutTs=1681268036403 2023-04-12 02:53:59,558 DEBUG [RS:1;jenkins-hbase4:44679-shortCompactions-0] compactions.Compactor(207): Compacting testSplitTableRegionUnevenDaughter=cc9012ec0b656baa24d04368fbe026c4-73d3880f14274287a94d30f14b9cf96e, keycount=10, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=71, earliestPutTs=1681268036642 2023-04-12 02:53:59,558 DEBUG [RS:1;jenkins-hbase4:44679-longCompactions-0] compactions.Compactor(207): Compacting testSplitTableRegionUnevenDaughter=cc9012ec0b656baa24d04368fbe026c4-260b76cda6d74251b52568d0821466a2, keycount=10, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=71, earliestPutTs=1681268036642 2023-04-12 02:53:59,559 DEBUG [RS:1;jenkins-hbase4:44679-shortCompactions-0] compactions.Compactor(207): Compacting testSplitTableRegionUnevenDaughter=cc9012ec0b656baa24d04368fbe026c4-5743ff156f3a424f80d4de48581aeeae, keycount=8, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=82, earliestPutTs=1681268036915 2023-04-12 02:53:59,559 DEBUG [RS:1;jenkins-hbase4:44679-longCompactions-0] compactions.Compactor(207): Compacting testSplitTableRegionUnevenDaughter=cc9012ec0b656baa24d04368fbe026c4-7ed05e97ddbc4e359abe117e7e8b3de9, keycount=8, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=82, earliestPutTs=1681268036915 2023-04-12 02:53:59,560 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=18, resume processing ppid=16 2023-04-12 02:53:59,560 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=18, ppid=16, state=SUCCESS; OpenRegionProcedure 6f55b5d6a460bd8c7569dee22f049104, server=jenkins-hbase4.apache.org,44679,1681268031772 in 536 msec 2023-04-12 02:53:59,566 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=16, ppid=12, state=SUCCESS; TransitRegionStateProcedure table=testSplitTableRegionUnevenDaughter, region=6f55b5d6a460bd8c7569dee22f049104, ASSIGN in 717 msec 2023-04-12 02:53:59,569 INFO [StoreOpener-e2a9b8fe15a781798c0b49e48ff5aad7-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf1 of region e2a9b8fe15a781798c0b49e48ff5aad7 2023-04-12 02:53:59,576 DEBUG [StoreOpener-e2a9b8fe15a781798c0b49e48ff5aad7-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/e2a9b8fe15a781798c0b49e48ff5aad7/cf1 2023-04-12 02:53:59,577 DEBUG [StoreOpener-e2a9b8fe15a781798c0b49e48ff5aad7-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/e2a9b8fe15a781798c0b49e48ff5aad7/cf1 2023-04-12 02:53:59,578 INFO [StoreOpener-e2a9b8fe15a781798c0b49e48ff5aad7-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e2a9b8fe15a781798c0b49e48ff5aad7 columnFamilyName cf1 2023-04-12 02:53:59,635 INFO [RS:1;jenkins-hbase4:44679-longCompactions-0] throttle.PressureAwareThroughputController(145): 6f55b5d6a460bd8c7569dee22f049104#cf2#compaction#14 average throughput is 0.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 02:53:59,647 DEBUG [StoreOpener-e2a9b8fe15a781798c0b49e48ff5aad7-1] regionserver.HStore(539): loaded hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/e2a9b8fe15a781798c0b49e48ff5aad7/cf1/07c3d7abf05f43cdbe44fe637da34a7d.cc9012ec0b656baa24d04368fbe026c4->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4/cf1/07c3d7abf05f43cdbe44fe637da34a7d-bottom 2023-04-12 02:53:59,659 INFO [RS:1;jenkins-hbase4:44679-shortCompactions-0] throttle.PressureAwareThroughputController(145): 6f55b5d6a460bd8c7569dee22f049104#cf1#compaction#15 average throughput is 0.43 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 02:53:59,667 DEBUG [StoreOpener-e2a9b8fe15a781798c0b49e48ff5aad7-1] regionserver.HStore(539): loaded hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/e2a9b8fe15a781798c0b49e48ff5aad7/cf1/364127255e644c3d8f0a55272e1ab83a.cc9012ec0b656baa24d04368fbe026c4->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4/cf1/364127255e644c3d8f0a55272e1ab83a-bottom 2023-04-12 02:53:59,677 DEBUG [StoreOpener-e2a9b8fe15a781798c0b49e48ff5aad7-1] regionserver.HStore(539): loaded hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/e2a9b8fe15a781798c0b49e48ff5aad7/cf1/a859f22056324f46889984d0e730b4bf.cc9012ec0b656baa24d04368fbe026c4->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4/cf1/a859f22056324f46889984d0e730b4bf-bottom 2023-04-12 02:53:59,683 DEBUG [StoreOpener-e2a9b8fe15a781798c0b49e48ff5aad7-1] regionserver.HStore(539): loaded hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/e2a9b8fe15a781798c0b49e48ff5aad7/cf1/b2cffa1e1e9f40ba81d73ca9452dfeb4.cc9012ec0b656baa24d04368fbe026c4->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4/cf1/b2cffa1e1e9f40ba81d73ca9452dfeb4-bottom 2023-04-12 02:53:59,691 INFO [StoreOpener-e2a9b8fe15a781798c0b49e48ff5aad7-1] regionserver.HStore(310): Store=e2a9b8fe15a781798c0b49e48ff5aad7/cf1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-12 02:53:59,692 INFO [StoreOpener-e2a9b8fe15a781798c0b49e48ff5aad7-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf2 of region e2a9b8fe15a781798c0b49e48ff5aad7 2023-04-12 02:53:59,693 DEBUG [StoreOpener-e2a9b8fe15a781798c0b49e48ff5aad7-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/e2a9b8fe15a781798c0b49e48ff5aad7/cf2 2023-04-12 02:53:59,693 DEBUG [StoreOpener-e2a9b8fe15a781798c0b49e48ff5aad7-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/e2a9b8fe15a781798c0b49e48ff5aad7/cf2 2023-04-12 02:53:59,694 INFO [StoreOpener-e2a9b8fe15a781798c0b49e48ff5aad7-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e2a9b8fe15a781798c0b49e48ff5aad7 columnFamilyName cf2 2023-04-12 02:53:59,756 DEBUG [StoreOpener-e2a9b8fe15a781798c0b49e48ff5aad7-1] regionserver.HStore(539): loaded hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/e2a9b8fe15a781798c0b49e48ff5aad7/cf2/18c4c7a41a6d44f9a7eb1d9d77806e17.cc9012ec0b656baa24d04368fbe026c4->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4/cf2/18c4c7a41a6d44f9a7eb1d9d77806e17-bottom 2023-04-12 02:53:59,756 DEBUG [RS:1;jenkins-hbase4:44679-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/6f55b5d6a460bd8c7569dee22f049104/.tmp/cf2/e3c99bab060c48aeaade535a58895af7 as hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/6f55b5d6a460bd8c7569dee22f049104/cf2/e3c99bab060c48aeaade535a58895af7 2023-04-12 02:53:59,757 DEBUG [RS:1;jenkins-hbase4:44679-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/6f55b5d6a460bd8c7569dee22f049104/.tmp/cf1/9529f8592dc64944803ca7ce757aed35 as hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/6f55b5d6a460bd8c7569dee22f049104/cf1/9529f8592dc64944803ca7ce757aed35 2023-04-12 02:53:59,777 DEBUG [StoreOpener-e2a9b8fe15a781798c0b49e48ff5aad7-1] regionserver.HStore(539): loaded hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/e2a9b8fe15a781798c0b49e48ff5aad7/cf2/1b7cac025fb940cab9783583b39cef46.cc9012ec0b656baa24d04368fbe026c4->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4/cf2/1b7cac025fb940cab9783583b39cef46-bottom 2023-04-12 02:53:59,783 INFO [RS:1;jenkins-hbase4:44679-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 6f55b5d6a460bd8c7569dee22f049104/cf1 of 6f55b5d6a460bd8c7569dee22f049104 into 9529f8592dc64944803ca7ce757aed35(size=6.7 K), total size for store is 6.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 02:53:59,786 DEBUG [RS:1;jenkins-hbase4:44679-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6f55b5d6a460bd8c7569dee22f049104: 2023-04-12 02:53:59,787 INFO [RS:1;jenkins-hbase4:44679-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=testSplitTableRegionUnevenDaughter,26,1681268036948.6f55b5d6a460bd8c7569dee22f049104., storeName=6f55b5d6a460bd8c7569dee22f049104/cf1, priority=9, startTime=1681268039526; duration=0sec 2023-04-12 02:53:59,787 DEBUG [RS:1;jenkins-hbase4:44679-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 02:53:59,795 INFO [RS:1;jenkins-hbase4:44679-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 6f55b5d6a460bd8c7569dee22f049104/cf2 of 6f55b5d6a460bd8c7569dee22f049104 into e3c99bab060c48aeaade535a58895af7(size=6.7 K), total size for store is 6.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 02:53:59,796 DEBUG [RS:1;jenkins-hbase4:44679-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6f55b5d6a460bd8c7569dee22f049104: 2023-04-12 02:53:59,796 INFO [RS:1;jenkins-hbase4:44679-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=testSplitTableRegionUnevenDaughter,26,1681268036948.6f55b5d6a460bd8c7569dee22f049104., storeName=6f55b5d6a460bd8c7569dee22f049104/cf2, priority=9, startTime=1681268039527; duration=0sec 2023-04-12 02:53:59,796 DEBUG [RS:1;jenkins-hbase4:44679-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 02:53:59,797 DEBUG [StoreOpener-e2a9b8fe15a781798c0b49e48ff5aad7-1] regionserver.HStore(539): loaded hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/e2a9b8fe15a781798c0b49e48ff5aad7/cf2/213df881d096427c9a5864569db2e1c4.cc9012ec0b656baa24d04368fbe026c4->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4/cf2/213df881d096427c9a5864569db2e1c4-bottom 2023-04-12 02:53:59,804 DEBUG [StoreOpener-e2a9b8fe15a781798c0b49e48ff5aad7-1] regionserver.HStore(539): loaded hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/e2a9b8fe15a781798c0b49e48ff5aad7/cf2/8df43cbe10c74ad8b2c721f3509bb1d1.cc9012ec0b656baa24d04368fbe026c4->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4/cf2/8df43cbe10c74ad8b2c721f3509bb1d1-bottom 2023-04-12 02:53:59,804 INFO [StoreOpener-e2a9b8fe15a781798c0b49e48ff5aad7-1] regionserver.HStore(310): Store=e2a9b8fe15a781798c0b49e48ff5aad7/cf2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-12 02:53:59,808 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/e2a9b8fe15a781798c0b49e48ff5aad7 2023-04-12 02:53:59,811 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/e2a9b8fe15a781798c0b49e48ff5aad7 2023-04-12 02:53:59,813 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testSplitTableRegionUnevenDaughter descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-04-12 02:53:59,815 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for e2a9b8fe15a781798c0b49e48ff5aad7 2023-04-12 02:53:59,816 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened e2a9b8fe15a781798c0b49e48ff5aad7; next sequenceid=86; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=74355014, jitterRate=0.1079760491847992}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-04-12 02:53:59,816 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for e2a9b8fe15a781798c0b49e48ff5aad7: 2023-04-12 02:53:59,817 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testSplitTableRegionUnevenDaughter,,1681268036948.e2a9b8fe15a781798c0b49e48ff5aad7., pid=17, masterSystemTime=1681268039161 2023-04-12 02:53:59,817 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: Opening Region; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 02:53:59,818 DEBUG [RS:1;jenkins-hbase4:44679-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-12 02:53:59,818 INFO [RS:1;jenkins-hbase4:44679-shortCompactions-0] regionserver.HStore(1898): Keeping/Overriding Compaction request priority to -2147482648 for CF cf1 since it belongs to recently split daughter region testSplitTableRegionUnevenDaughter,,1681268036948.e2a9b8fe15a781798c0b49e48ff5aad7. 2023-04-12 02:53:59,818 DEBUG [RS:1;jenkins-hbase4:44679-shortCompactions-0] regionserver.HStore(1912): e2a9b8fe15a781798c0b49e48ff5aad7/cf1 is initiating minor compaction (all files) 2023-04-12 02:53:59,819 INFO [RS:1;jenkins-hbase4:44679-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of e2a9b8fe15a781798c0b49e48ff5aad7/cf1 in testSplitTableRegionUnevenDaughter,,1681268036948.e2a9b8fe15a781798c0b49e48ff5aad7. 2023-04-12 02:53:59,819 INFO [RS:1;jenkins-hbase4:44679-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/e2a9b8fe15a781798c0b49e48ff5aad7/cf1/b2cffa1e1e9f40ba81d73ca9452dfeb4.cc9012ec0b656baa24d04368fbe026c4->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4/cf1/b2cffa1e1e9f40ba81d73ca9452dfeb4-bottom, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/e2a9b8fe15a781798c0b49e48ff5aad7/cf1/07c3d7abf05f43cdbe44fe637da34a7d.cc9012ec0b656baa24d04368fbe026c4->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4/cf1/07c3d7abf05f43cdbe44fe637da34a7d-bottom, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/e2a9b8fe15a781798c0b49e48ff5aad7/cf1/364127255e644c3d8f0a55272e1ab83a.cc9012ec0b656baa24d04368fbe026c4->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4/cf1/364127255e644c3d8f0a55272e1ab83a-bottom, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/e2a9b8fe15a781798c0b49e48ff5aad7/cf1/a859f22056324f46889984d0e730b4bf.cc9012ec0b656baa24d04368fbe026c4->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4/cf1/a859f22056324f46889984d0e730b4bf-bottom] into tmpdir=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/e2a9b8fe15a781798c0b49e48ff5aad7/.tmp, totalSize=19.9 K 2023-04-12 02:53:59,819 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: Opening Region; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 02:53:59,820 DEBUG [RS:1;jenkins-hbase4:44679-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-12 02:53:59,821 INFO [RS:1;jenkins-hbase4:44679-longCompactions-0] regionserver.HStore(1898): Keeping/Overriding Compaction request priority to -2147482648 for CF cf2 since it belongs to recently split daughter region testSplitTableRegionUnevenDaughter,,1681268036948.e2a9b8fe15a781798c0b49e48ff5aad7. 2023-04-12 02:53:59,821 DEBUG [RS:1;jenkins-hbase4:44679-shortCompactions-0] compactions.Compactor(207): Compacting b2cffa1e1e9f40ba81d73ca9452dfeb4.cc9012ec0b656baa24d04368fbe026c4, keycount=1, bloomtype=ROW, size=4.8 K, encoding=NONE, compression=NONE, seqNum=6, earliestPutTs=1681268035253 2023-04-12 02:53:59,821 DEBUG [RS:1;jenkins-hbase4:44679-longCompactions-0] regionserver.HStore(1912): e2a9b8fe15a781798c0b49e48ff5aad7/cf2 is initiating minor compaction (all files) 2023-04-12 02:53:59,821 INFO [RS:1;jenkins-hbase4:44679-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e2a9b8fe15a781798c0b49e48ff5aad7/cf2 in testSplitTableRegionUnevenDaughter,,1681268036948.e2a9b8fe15a781798c0b49e48ff5aad7. 2023-04-12 02:53:59,822 DEBUG [RS:1;jenkins-hbase4:44679-shortCompactions-0] compactions.Compactor(207): Compacting 07c3d7abf05f43cdbe44fe637da34a7d.cc9012ec0b656baa24d04368fbe026c4, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=19, earliestPutTs=1681268035652 2023-04-12 02:53:59,822 INFO [RS:1;jenkins-hbase4:44679-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/e2a9b8fe15a781798c0b49e48ff5aad7/cf2/18c4c7a41a6d44f9a7eb1d9d77806e17.cc9012ec0b656baa24d04368fbe026c4->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4/cf2/18c4c7a41a6d44f9a7eb1d9d77806e17-bottom, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/e2a9b8fe15a781798c0b49e48ff5aad7/cf2/8df43cbe10c74ad8b2c721f3509bb1d1.cc9012ec0b656baa24d04368fbe026c4->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4/cf2/8df43cbe10c74ad8b2c721f3509bb1d1-bottom, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/e2a9b8fe15a781798c0b49e48ff5aad7/cf2/1b7cac025fb940cab9783583b39cef46.cc9012ec0b656baa24d04368fbe026c4->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4/cf2/1b7cac025fb940cab9783583b39cef46-bottom, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/e2a9b8fe15a781798c0b49e48ff5aad7/cf2/213df881d096427c9a5864569db2e1c4.cc9012ec0b656baa24d04368fbe026c4->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4/cf2/213df881d096427c9a5864569db2e1c4-bottom] into tmpdir=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/e2a9b8fe15a781798c0b49e48ff5aad7/.tmp, totalSize=19.9 K 2023-04-12 02:53:59,822 DEBUG [RS:1;jenkins-hbase4:44679-longCompactions-0] compactions.Compactor(207): Compacting 18c4c7a41a6d44f9a7eb1d9d77806e17.cc9012ec0b656baa24d04368fbe026c4, keycount=1, bloomtype=ROW, size=4.8 K, encoding=NONE, compression=NONE, seqNum=6, earliestPutTs=1681268035253 2023-04-12 02:53:59,822 DEBUG [RS:1;jenkins-hbase4:44679-shortCompactions-0] compactions.Compactor(207): Compacting 364127255e644c3d8f0a55272e1ab83a.cc9012ec0b656baa24d04368fbe026c4, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=32, earliestPutTs=1681268035903 2023-04-12 02:53:59,823 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testSplitTableRegionUnevenDaughter,,1681268036948.e2a9b8fe15a781798c0b49e48ff5aad7. 2023-04-12 02:53:59,823 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testSplitTableRegionUnevenDaughter,,1681268036948.e2a9b8fe15a781798c0b49e48ff5aad7. 2023-04-12 02:53:59,824 DEBUG [RS:1;jenkins-hbase4:44679-longCompactions-0] compactions.Compactor(207): Compacting 8df43cbe10c74ad8b2c721f3509bb1d1.cc9012ec0b656baa24d04368fbe026c4, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=19, earliestPutTs=1681268035652 2023-04-12 02:53:59,824 DEBUG [RS:1;jenkins-hbase4:44679-shortCompactions-0] compactions.Compactor(207): Compacting a859f22056324f46889984d0e730b4bf.cc9012ec0b656baa24d04368fbe026c4, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=45, earliestPutTs=1681268036147 2023-04-12 02:53:59,825 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=15 updating hbase:meta row=e2a9b8fe15a781798c0b49e48ff5aad7, regionState=OPEN, openSeqNum=86, regionLocation=jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:53:59,825 DEBUG [RS:1;jenkins-hbase4:44679-longCompactions-0] compactions.Compactor(207): Compacting 1b7cac025fb940cab9783583b39cef46.cc9012ec0b656baa24d04368fbe026c4, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=32, earliestPutTs=1681268035903 2023-04-12 02:53:59,825 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testSplitTableRegionUnevenDaughter,,1681268036948.e2a9b8fe15a781798c0b49e48ff5aad7.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1681268039825"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1681268039825"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681268039825"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681268039825"}]},"ts":"1681268039825"} 2023-04-12 02:53:59,826 DEBUG [RS:1;jenkins-hbase4:44679-longCompactions-0] compactions.Compactor(207): Compacting 213df881d096427c9a5864569db2e1c4.cc9012ec0b656baa24d04368fbe026c4, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=45, earliestPutTs=1681268036147 2023-04-12 02:53:59,834 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=17, resume processing ppid=15 2023-04-12 02:53:59,836 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=17, ppid=15, state=SUCCESS; OpenRegionProcedure e2a9b8fe15a781798c0b49e48ff5aad7, server=jenkins-hbase4.apache.org,44679,1681268031772 in 824 msec 2023-04-12 02:53:59,839 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=15, resume processing ppid=12 2023-04-12 02:53:59,839 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=15, ppid=12, state=SUCCESS; TransitRegionStateProcedure table=testSplitTableRegionUnevenDaughter, region=e2a9b8fe15a781798c0b49e48ff5aad7, ASSIGN in 993 msec 2023-04-12 02:53:59,842 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=cc9012ec0b656baa24d04368fbe026c4, daughterA=e2a9b8fe15a781798c0b49e48ff5aad7, daughterB=6f55b5d6a460bd8c7569dee22f049104 in 2.8910 sec 2023-04-12 02:53:59,850 INFO [RS:1;jenkins-hbase4:44679-shortCompactions-0] throttle.PressureAwareThroughputController(145): e2a9b8fe15a781798c0b49e48ff5aad7#cf1#compaction#16 average throughput is 0.43 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 02:53:59,859 INFO [RS:1;jenkins-hbase4:44679-longCompactions-0] throttle.PressureAwareThroughputController(145): e2a9b8fe15a781798c0b49e48ff5aad7#cf2#compaction#17 average throughput is 0.43 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 02:53:59,918 DEBUG [RS:1;jenkins-hbase4:44679-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/e2a9b8fe15a781798c0b49e48ff5aad7/.tmp/cf1/6bf68177d71a430692424060c2986643 as hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/e2a9b8fe15a781798c0b49e48ff5aad7/cf1/6bf68177d71a430692424060c2986643 2023-04-12 02:53:59,927 INFO [RS:1;jenkins-hbase4:44679-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in e2a9b8fe15a781798c0b49e48ff5aad7/cf1 of e2a9b8fe15a781798c0b49e48ff5aad7 into 6bf68177d71a430692424060c2986643(size=5.5 K), total size for store is 5.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 02:53:59,927 DEBUG [RS:1;jenkins-hbase4:44679-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for e2a9b8fe15a781798c0b49e48ff5aad7: 2023-04-12 02:53:59,927 INFO [RS:1;jenkins-hbase4:44679-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=testSplitTableRegionUnevenDaughter,,1681268036948.e2a9b8fe15a781798c0b49e48ff5aad7., storeName=e2a9b8fe15a781798c0b49e48ff5aad7/cf1, priority=12, startTime=1681268039817; duration=0sec 2023-04-12 02:53:59,927 DEBUG [RS:1;jenkins-hbase4:44679-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 02:53:59,930 DEBUG [RS:1;jenkins-hbase4:44679-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/e2a9b8fe15a781798c0b49e48ff5aad7/.tmp/cf2/67d6621e91af49eaab0011c0742d2c28 as hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/e2a9b8fe15a781798c0b49e48ff5aad7/cf2/67d6621e91af49eaab0011c0742d2c28 2023-04-12 02:53:59,946 INFO [RS:1;jenkins-hbase4:44679-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in e2a9b8fe15a781798c0b49e48ff5aad7/cf2 of e2a9b8fe15a781798c0b49e48ff5aad7 into 67d6621e91af49eaab0011c0742d2c28(size=5.5 K), total size for store is 5.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 02:53:59,947 DEBUG [RS:1;jenkins-hbase4:44679-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e2a9b8fe15a781798c0b49e48ff5aad7: 2023-04-12 02:53:59,947 INFO [RS:1;jenkins-hbase4:44679-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=testSplitTableRegionUnevenDaughter,,1681268036948.e2a9b8fe15a781798c0b49e48ff5aad7., storeName=e2a9b8fe15a781798c0b49e48ff5aad7/cf2, priority=12, startTime=1681268039819; duration=0sec 2023-04-12 02:53:59,947 DEBUG [RS:1;jenkins-hbase4:44679-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 02:53:59,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44679] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Scan size: 157 connection: 172.31.14.131:43794 deadline: 1681268099962, exception=org.apache.hadoop.hbase.NotServingRegionException: testSplitTableRegionUnevenDaughter,,1681268034745.cc9012ec0b656baa24d04368fbe026c4. is not online on jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:00,126 DEBUG [Listener at localhost/46667] client.ClientScanner(240): Advancing internal scanner to startKey at '26', inclusive 2023-04-12 02:54:00,155 INFO [Listener at localhost/46667] assignment.TestSplitTableRegionProcedure(570): Row Count = 60 2023-04-12 02:54:00,158 DEBUG [Listener at localhost/46667] client.ClientScanner(240): Advancing internal scanner to startKey at '26', inclusive 2023-04-12 02:54:00,218 WARN [Listener at localhost/46667] procedure2.ProcedureTestingUtility(193): Set Kill before store update to: false 2023-04-12 02:54:00,236 INFO [Listener at localhost/46667] client.HBaseAdmin$15(890): Started disable of testSplitTableRegionUnevenDaughter 2023-04-12 02:54:00,242 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40281] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable testSplitTableRegionUnevenDaughter 2023-04-12 02:54:00,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40281] procedure2.ProcedureExecutor(1029): Stored pid=19, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-12 02:54:00,265 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testSplitTableRegionUnevenDaughter","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681268040265"}]},"ts":"1681268040265"} 2023-04-12 02:54:00,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] master.MasterRpcServices(1227): Checking to see if procedure is done pid=19 2023-04-12 02:54:00,269 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testSplitTableRegionUnevenDaughter, state=DISABLING in hbase:meta 2023-04-12 02:54:00,271 INFO [PEWorker-1] procedure.DisableTableProcedure(293): Set testSplitTableRegionUnevenDaughter to state=DISABLING 2023-04-12 02:54:00,274 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=20, ppid=19, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitTableRegionUnevenDaughter, region=e2a9b8fe15a781798c0b49e48ff5aad7, UNASSIGN}, {pid=21, ppid=19, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitTableRegionUnevenDaughter, region=6f55b5d6a460bd8c7569dee22f049104, UNASSIGN}] 2023-04-12 02:54:00,275 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=21, ppid=19, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitTableRegionUnevenDaughter, region=6f55b5d6a460bd8c7569dee22f049104, UNASSIGN 2023-04-12 02:54:00,276 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=6f55b5d6a460bd8c7569dee22f049104, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:00,277 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitTableRegionUnevenDaughter,26,1681268036948.6f55b5d6a460bd8c7569dee22f049104.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1681268040276"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681268040276"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681268040276"}]},"ts":"1681268040276"} 2023-04-12 02:54:00,279 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=22, ppid=21, state=RUNNABLE; CloseRegionProcedure 6f55b5d6a460bd8c7569dee22f049104, server=jenkins-hbase4.apache.org,44679,1681268031772}] 2023-04-12 02:54:00,289 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=20, ppid=19, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitTableRegionUnevenDaughter, region=e2a9b8fe15a781798c0b49e48ff5aad7, UNASSIGN 2023-04-12 02:54:00,290 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=20 updating hbase:meta row=e2a9b8fe15a781798c0b49e48ff5aad7, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:00,291 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitTableRegionUnevenDaughter,,1681268036948.e2a9b8fe15a781798c0b49e48ff5aad7.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1681268040290"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681268040290"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681268040290"}]},"ts":"1681268040290"} 2023-04-12 02:54:00,293 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=23, ppid=20, state=RUNNABLE; CloseRegionProcedure e2a9b8fe15a781798c0b49e48ff5aad7, server=jenkins-hbase4.apache.org,44679,1681268031772}] 2023-04-12 02:54:00,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] master.MasterRpcServices(1227): Checking to see if procedure is done pid=19 2023-04-12 02:54:00,433 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 6f55b5d6a460bd8c7569dee22f049104 2023-04-12 02:54:00,433 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 6f55b5d6a460bd8c7569dee22f049104, disabling compactions & flushes 2023-04-12 02:54:00,433 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region testSplitTableRegionUnevenDaughter,26,1681268036948.6f55b5d6a460bd8c7569dee22f049104. 2023-04-12 02:54:00,433 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testSplitTableRegionUnevenDaughter,26,1681268036948.6f55b5d6a460bd8c7569dee22f049104. 2023-04-12 02:54:00,433 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on testSplitTableRegionUnevenDaughter,26,1681268036948.6f55b5d6a460bd8c7569dee22f049104. after waiting 0 ms 2023-04-12 02:54:00,433 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region testSplitTableRegionUnevenDaughter,26,1681268036948.6f55b5d6a460bd8c7569dee22f049104. 2023-04-12 02:54:00,456 DEBUG [StoreCloser-testSplitTableRegionUnevenDaughter,26,1681268036948.6f55b5d6a460bd8c7569dee22f049104.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/6f55b5d6a460bd8c7569dee22f049104/cf1/b2cffa1e1e9f40ba81d73ca9452dfeb4.cc9012ec0b656baa24d04368fbe026c4->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4/cf1/b2cffa1e1e9f40ba81d73ca9452dfeb4-top, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/6f55b5d6a460bd8c7569dee22f049104/cf1/07c3d7abf05f43cdbe44fe637da34a7d.cc9012ec0b656baa24d04368fbe026c4->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4/cf1/07c3d7abf05f43cdbe44fe637da34a7d-top, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/6f55b5d6a460bd8c7569dee22f049104/cf1/364127255e644c3d8f0a55272e1ab83a.cc9012ec0b656baa24d04368fbe026c4->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4/cf1/364127255e644c3d8f0a55272e1ab83a-top, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/6f55b5d6a460bd8c7569dee22f049104/cf1/a859f22056324f46889984d0e730b4bf.cc9012ec0b656baa24d04368fbe026c4->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4/cf1/a859f22056324f46889984d0e730b4bf-top, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/6f55b5d6a460bd8c7569dee22f049104/cf1/testSplitTableRegionUnevenDaughter=cc9012ec0b656baa24d04368fbe026c4-a32265ffc17446ddb889acfdd1472f9c, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/6f55b5d6a460bd8c7569dee22f049104/cf1/testSplitTableRegionUnevenDaughter=cc9012ec0b656baa24d04368fbe026c4-73d3880f14274287a94d30f14b9cf96e, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/6f55b5d6a460bd8c7569dee22f049104/cf1/testSplitTableRegionUnevenDaughter=cc9012ec0b656baa24d04368fbe026c4-5743ff156f3a424f80d4de48581aeeae] to archive 2023-04-12 02:54:00,458 DEBUG [StoreCloser-testSplitTableRegionUnevenDaughter,26,1681268036948.6f55b5d6a460bd8c7569dee22f049104.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-12 02:54:00,484 DEBUG [StoreCloser-testSplitTableRegionUnevenDaughter,26,1681268036948.6f55b5d6a460bd8c7569dee22f049104.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/6f55b5d6a460bd8c7569dee22f049104/cf1/b2cffa1e1e9f40ba81d73ca9452dfeb4.cc9012ec0b656baa24d04368fbe026c4 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegionUnevenDaughter/6f55b5d6a460bd8c7569dee22f049104/cf1/b2cffa1e1e9f40ba81d73ca9452dfeb4.cc9012ec0b656baa24d04368fbe026c4 2023-04-12 02:54:00,496 DEBUG [StoreCloser-testSplitTableRegionUnevenDaughter,26,1681268036948.6f55b5d6a460bd8c7569dee22f049104.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/6f55b5d6a460bd8c7569dee22f049104/cf1/07c3d7abf05f43cdbe44fe637da34a7d.cc9012ec0b656baa24d04368fbe026c4 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegionUnevenDaughter/6f55b5d6a460bd8c7569dee22f049104/cf1/07c3d7abf05f43cdbe44fe637da34a7d.cc9012ec0b656baa24d04368fbe026c4 2023-04-12 02:54:00,502 DEBUG [StoreCloser-testSplitTableRegionUnevenDaughter,26,1681268036948.6f55b5d6a460bd8c7569dee22f049104.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/6f55b5d6a460bd8c7569dee22f049104/cf1/364127255e644c3d8f0a55272e1ab83a.cc9012ec0b656baa24d04368fbe026c4 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegionUnevenDaughter/6f55b5d6a460bd8c7569dee22f049104/cf1/364127255e644c3d8f0a55272e1ab83a.cc9012ec0b656baa24d04368fbe026c4 2023-04-12 02:54:00,504 DEBUG [StoreCloser-testSplitTableRegionUnevenDaughter,26,1681268036948.6f55b5d6a460bd8c7569dee22f049104.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/6f55b5d6a460bd8c7569dee22f049104/cf1/a859f22056324f46889984d0e730b4bf.cc9012ec0b656baa24d04368fbe026c4 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegionUnevenDaughter/6f55b5d6a460bd8c7569dee22f049104/cf1/a859f22056324f46889984d0e730b4bf.cc9012ec0b656baa24d04368fbe026c4 2023-04-12 02:54:00,506 DEBUG [StoreCloser-testSplitTableRegionUnevenDaughter,26,1681268036948.6f55b5d6a460bd8c7569dee22f049104.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/6f55b5d6a460bd8c7569dee22f049104/cf1/testSplitTableRegionUnevenDaughter=cc9012ec0b656baa24d04368fbe026c4-a32265ffc17446ddb889acfdd1472f9c to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegionUnevenDaughter/6f55b5d6a460bd8c7569dee22f049104/cf1/testSplitTableRegionUnevenDaughter=cc9012ec0b656baa24d04368fbe026c4-a32265ffc17446ddb889acfdd1472f9c 2023-04-12 02:54:00,508 DEBUG [StoreCloser-testSplitTableRegionUnevenDaughter,26,1681268036948.6f55b5d6a460bd8c7569dee22f049104.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/6f55b5d6a460bd8c7569dee22f049104/cf1/testSplitTableRegionUnevenDaughter=cc9012ec0b656baa24d04368fbe026c4-73d3880f14274287a94d30f14b9cf96e to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegionUnevenDaughter/6f55b5d6a460bd8c7569dee22f049104/cf1/testSplitTableRegionUnevenDaughter=cc9012ec0b656baa24d04368fbe026c4-73d3880f14274287a94d30f14b9cf96e 2023-04-12 02:54:00,511 DEBUG [StoreCloser-testSplitTableRegionUnevenDaughter,26,1681268036948.6f55b5d6a460bd8c7569dee22f049104.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/6f55b5d6a460bd8c7569dee22f049104/cf1/testSplitTableRegionUnevenDaughter=cc9012ec0b656baa24d04368fbe026c4-5743ff156f3a424f80d4de48581aeeae to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegionUnevenDaughter/6f55b5d6a460bd8c7569dee22f049104/cf1/testSplitTableRegionUnevenDaughter=cc9012ec0b656baa24d04368fbe026c4-5743ff156f3a424f80d4de48581aeeae 2023-04-12 02:54:00,544 DEBUG [StoreCloser-testSplitTableRegionUnevenDaughter,26,1681268036948.6f55b5d6a460bd8c7569dee22f049104.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/6f55b5d6a460bd8c7569dee22f049104/cf2/18c4c7a41a6d44f9a7eb1d9d77806e17.cc9012ec0b656baa24d04368fbe026c4->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4/cf2/18c4c7a41a6d44f9a7eb1d9d77806e17-top, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/6f55b5d6a460bd8c7569dee22f049104/cf2/8df43cbe10c74ad8b2c721f3509bb1d1.cc9012ec0b656baa24d04368fbe026c4->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4/cf2/8df43cbe10c74ad8b2c721f3509bb1d1-top, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/6f55b5d6a460bd8c7569dee22f049104/cf2/1b7cac025fb940cab9783583b39cef46.cc9012ec0b656baa24d04368fbe026c4->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4/cf2/1b7cac025fb940cab9783583b39cef46-top, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/6f55b5d6a460bd8c7569dee22f049104/cf2/213df881d096427c9a5864569db2e1c4.cc9012ec0b656baa24d04368fbe026c4->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4/cf2/213df881d096427c9a5864569db2e1c4-top, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/6f55b5d6a460bd8c7569dee22f049104/cf2/testSplitTableRegionUnevenDaughter=cc9012ec0b656baa24d04368fbe026c4-3310af0091c24e11ae1d57719b30d33b, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/6f55b5d6a460bd8c7569dee22f049104/cf2/testSplitTableRegionUnevenDaughter=cc9012ec0b656baa24d04368fbe026c4-260b76cda6d74251b52568d0821466a2, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/6f55b5d6a460bd8c7569dee22f049104/cf2/testSplitTableRegionUnevenDaughter=cc9012ec0b656baa24d04368fbe026c4-7ed05e97ddbc4e359abe117e7e8b3de9] to archive 2023-04-12 02:54:00,545 DEBUG [StoreCloser-testSplitTableRegionUnevenDaughter,26,1681268036948.6f55b5d6a460bd8c7569dee22f049104.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-12 02:54:00,548 DEBUG [StoreCloser-testSplitTableRegionUnevenDaughter,26,1681268036948.6f55b5d6a460bd8c7569dee22f049104.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/6f55b5d6a460bd8c7569dee22f049104/cf2/18c4c7a41a6d44f9a7eb1d9d77806e17.cc9012ec0b656baa24d04368fbe026c4 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegionUnevenDaughter/6f55b5d6a460bd8c7569dee22f049104/cf2/18c4c7a41a6d44f9a7eb1d9d77806e17.cc9012ec0b656baa24d04368fbe026c4 2023-04-12 02:54:00,550 DEBUG [StoreCloser-testSplitTableRegionUnevenDaughter,26,1681268036948.6f55b5d6a460bd8c7569dee22f049104.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/6f55b5d6a460bd8c7569dee22f049104/cf2/8df43cbe10c74ad8b2c721f3509bb1d1.cc9012ec0b656baa24d04368fbe026c4 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegionUnevenDaughter/6f55b5d6a460bd8c7569dee22f049104/cf2/8df43cbe10c74ad8b2c721f3509bb1d1.cc9012ec0b656baa24d04368fbe026c4 2023-04-12 02:54:00,551 DEBUG [StoreCloser-testSplitTableRegionUnevenDaughter,26,1681268036948.6f55b5d6a460bd8c7569dee22f049104.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/6f55b5d6a460bd8c7569dee22f049104/cf2/1b7cac025fb940cab9783583b39cef46.cc9012ec0b656baa24d04368fbe026c4 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegionUnevenDaughter/6f55b5d6a460bd8c7569dee22f049104/cf2/1b7cac025fb940cab9783583b39cef46.cc9012ec0b656baa24d04368fbe026c4 2023-04-12 02:54:00,553 DEBUG [StoreCloser-testSplitTableRegionUnevenDaughter,26,1681268036948.6f55b5d6a460bd8c7569dee22f049104.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/6f55b5d6a460bd8c7569dee22f049104/cf2/213df881d096427c9a5864569db2e1c4.cc9012ec0b656baa24d04368fbe026c4 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegionUnevenDaughter/6f55b5d6a460bd8c7569dee22f049104/cf2/213df881d096427c9a5864569db2e1c4.cc9012ec0b656baa24d04368fbe026c4 2023-04-12 02:54:00,557 DEBUG [StoreCloser-testSplitTableRegionUnevenDaughter,26,1681268036948.6f55b5d6a460bd8c7569dee22f049104.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/6f55b5d6a460bd8c7569dee22f049104/cf2/testSplitTableRegionUnevenDaughter=cc9012ec0b656baa24d04368fbe026c4-3310af0091c24e11ae1d57719b30d33b to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegionUnevenDaughter/6f55b5d6a460bd8c7569dee22f049104/cf2/testSplitTableRegionUnevenDaughter=cc9012ec0b656baa24d04368fbe026c4-3310af0091c24e11ae1d57719b30d33b 2023-04-12 02:54:00,560 DEBUG [StoreCloser-testSplitTableRegionUnevenDaughter,26,1681268036948.6f55b5d6a460bd8c7569dee22f049104.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/6f55b5d6a460bd8c7569dee22f049104/cf2/testSplitTableRegionUnevenDaughter=cc9012ec0b656baa24d04368fbe026c4-260b76cda6d74251b52568d0821466a2 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegionUnevenDaughter/6f55b5d6a460bd8c7569dee22f049104/cf2/testSplitTableRegionUnevenDaughter=cc9012ec0b656baa24d04368fbe026c4-260b76cda6d74251b52568d0821466a2 2023-04-12 02:54:00,561 DEBUG [StoreCloser-testSplitTableRegionUnevenDaughter,26,1681268036948.6f55b5d6a460bd8c7569dee22f049104.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/6f55b5d6a460bd8c7569dee22f049104/cf2/testSplitTableRegionUnevenDaughter=cc9012ec0b656baa24d04368fbe026c4-7ed05e97ddbc4e359abe117e7e8b3de9 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegionUnevenDaughter/6f55b5d6a460bd8c7569dee22f049104/cf2/testSplitTableRegionUnevenDaughter=cc9012ec0b656baa24d04368fbe026c4-7ed05e97ddbc4e359abe117e7e8b3de9 2023-04-12 02:54:00,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] master.MasterRpcServices(1227): Checking to see if procedure is done pid=19 2023-04-12 02:54:00,583 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/6f55b5d6a460bd8c7569dee22f049104/recovered.edits/90.seqid, newMaxSeqId=90, maxSeqId=85 2023-04-12 02:54:00,586 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro 2023-04-12 02:54:00,588 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed testSplitTableRegionUnevenDaughter,26,1681268036948.6f55b5d6a460bd8c7569dee22f049104. 2023-04-12 02:54:00,588 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 6f55b5d6a460bd8c7569dee22f049104: 2023-04-12 02:54:00,593 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 6f55b5d6a460bd8c7569dee22f049104 2023-04-12 02:54:00,593 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close e2a9b8fe15a781798c0b49e48ff5aad7 2023-04-12 02:54:00,594 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing e2a9b8fe15a781798c0b49e48ff5aad7, disabling compactions & flushes 2023-04-12 02:54:00,594 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region testSplitTableRegionUnevenDaughter,,1681268036948.e2a9b8fe15a781798c0b49e48ff5aad7. 2023-04-12 02:54:00,594 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testSplitTableRegionUnevenDaughter,,1681268036948.e2a9b8fe15a781798c0b49e48ff5aad7. 2023-04-12 02:54:00,594 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=6f55b5d6a460bd8c7569dee22f049104, regionState=CLOSED 2023-04-12 02:54:00,594 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on testSplitTableRegionUnevenDaughter,,1681268036948.e2a9b8fe15a781798c0b49e48ff5aad7. after waiting 0 ms 2023-04-12 02:54:00,594 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region testSplitTableRegionUnevenDaughter,,1681268036948.e2a9b8fe15a781798c0b49e48ff5aad7. 2023-04-12 02:54:00,594 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testSplitTableRegionUnevenDaughter,26,1681268036948.6f55b5d6a460bd8c7569dee22f049104.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1681268040594"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681268040594"}]},"ts":"1681268040594"} 2023-04-12 02:54:00,595 DEBUG [StoreCloser-testSplitTableRegionUnevenDaughter,,1681268036948.e2a9b8fe15a781798c0b49e48ff5aad7.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/e2a9b8fe15a781798c0b49e48ff5aad7/cf1/b2cffa1e1e9f40ba81d73ca9452dfeb4.cc9012ec0b656baa24d04368fbe026c4->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4/cf1/b2cffa1e1e9f40ba81d73ca9452dfeb4-bottom, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/e2a9b8fe15a781798c0b49e48ff5aad7/cf1/07c3d7abf05f43cdbe44fe637da34a7d.cc9012ec0b656baa24d04368fbe026c4->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4/cf1/07c3d7abf05f43cdbe44fe637da34a7d-bottom, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/e2a9b8fe15a781798c0b49e48ff5aad7/cf1/364127255e644c3d8f0a55272e1ab83a.cc9012ec0b656baa24d04368fbe026c4->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4/cf1/364127255e644c3d8f0a55272e1ab83a-bottom, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/e2a9b8fe15a781798c0b49e48ff5aad7/cf1/a859f22056324f46889984d0e730b4bf.cc9012ec0b656baa24d04368fbe026c4->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4/cf1/a859f22056324f46889984d0e730b4bf-bottom] to archive 2023-04-12 02:54:00,597 DEBUG [StoreCloser-testSplitTableRegionUnevenDaughter,,1681268036948.e2a9b8fe15a781798c0b49e48ff5aad7.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-12 02:54:00,606 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=22, resume processing ppid=21 2023-04-12 02:54:00,606 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=22, ppid=21, state=SUCCESS; CloseRegionProcedure 6f55b5d6a460bd8c7569dee22f049104, server=jenkins-hbase4.apache.org,44679,1681268031772 in 321 msec 2023-04-12 02:54:00,608 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=21, ppid=19, state=SUCCESS; TransitRegionStateProcedure table=testSplitTableRegionUnevenDaughter, region=6f55b5d6a460bd8c7569dee22f049104, UNASSIGN in 333 msec 2023-04-12 02:54:00,610 DEBUG [StoreCloser-testSplitTableRegionUnevenDaughter,,1681268036948.e2a9b8fe15a781798c0b49e48ff5aad7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/e2a9b8fe15a781798c0b49e48ff5aad7/cf1/b2cffa1e1e9f40ba81d73ca9452dfeb4.cc9012ec0b656baa24d04368fbe026c4 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegionUnevenDaughter/e2a9b8fe15a781798c0b49e48ff5aad7/cf1/b2cffa1e1e9f40ba81d73ca9452dfeb4.cc9012ec0b656baa24d04368fbe026c4 2023-04-12 02:54:00,613 DEBUG [StoreCloser-testSplitTableRegionUnevenDaughter,,1681268036948.e2a9b8fe15a781798c0b49e48ff5aad7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/e2a9b8fe15a781798c0b49e48ff5aad7/cf1/07c3d7abf05f43cdbe44fe637da34a7d.cc9012ec0b656baa24d04368fbe026c4 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegionUnevenDaughter/e2a9b8fe15a781798c0b49e48ff5aad7/cf1/07c3d7abf05f43cdbe44fe637da34a7d.cc9012ec0b656baa24d04368fbe026c4 2023-04-12 02:54:00,615 DEBUG [StoreCloser-testSplitTableRegionUnevenDaughter,,1681268036948.e2a9b8fe15a781798c0b49e48ff5aad7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/e2a9b8fe15a781798c0b49e48ff5aad7/cf1/364127255e644c3d8f0a55272e1ab83a.cc9012ec0b656baa24d04368fbe026c4 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegionUnevenDaughter/e2a9b8fe15a781798c0b49e48ff5aad7/cf1/364127255e644c3d8f0a55272e1ab83a.cc9012ec0b656baa24d04368fbe026c4 2023-04-12 02:54:00,617 DEBUG [StoreCloser-testSplitTableRegionUnevenDaughter,,1681268036948.e2a9b8fe15a781798c0b49e48ff5aad7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/e2a9b8fe15a781798c0b49e48ff5aad7/cf1/a859f22056324f46889984d0e730b4bf.cc9012ec0b656baa24d04368fbe026c4 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegionUnevenDaughter/e2a9b8fe15a781798c0b49e48ff5aad7/cf1/a859f22056324f46889984d0e730b4bf.cc9012ec0b656baa24d04368fbe026c4 2023-04-12 02:54:00,642 DEBUG [StoreCloser-testSplitTableRegionUnevenDaughter,,1681268036948.e2a9b8fe15a781798c0b49e48ff5aad7.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/e2a9b8fe15a781798c0b49e48ff5aad7/cf2/18c4c7a41a6d44f9a7eb1d9d77806e17.cc9012ec0b656baa24d04368fbe026c4->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4/cf2/18c4c7a41a6d44f9a7eb1d9d77806e17-bottom, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/e2a9b8fe15a781798c0b49e48ff5aad7/cf2/8df43cbe10c74ad8b2c721f3509bb1d1.cc9012ec0b656baa24d04368fbe026c4->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4/cf2/8df43cbe10c74ad8b2c721f3509bb1d1-bottom, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/e2a9b8fe15a781798c0b49e48ff5aad7/cf2/1b7cac025fb940cab9783583b39cef46.cc9012ec0b656baa24d04368fbe026c4->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4/cf2/1b7cac025fb940cab9783583b39cef46-bottom, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/e2a9b8fe15a781798c0b49e48ff5aad7/cf2/213df881d096427c9a5864569db2e1c4.cc9012ec0b656baa24d04368fbe026c4->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4/cf2/213df881d096427c9a5864569db2e1c4-bottom] to archive 2023-04-12 02:54:00,643 DEBUG [StoreCloser-testSplitTableRegionUnevenDaughter,,1681268036948.e2a9b8fe15a781798c0b49e48ff5aad7.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-12 02:54:00,650 DEBUG [StoreCloser-testSplitTableRegionUnevenDaughter,,1681268036948.e2a9b8fe15a781798c0b49e48ff5aad7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/e2a9b8fe15a781798c0b49e48ff5aad7/cf2/18c4c7a41a6d44f9a7eb1d9d77806e17.cc9012ec0b656baa24d04368fbe026c4 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegionUnevenDaughter/e2a9b8fe15a781798c0b49e48ff5aad7/cf2/18c4c7a41a6d44f9a7eb1d9d77806e17.cc9012ec0b656baa24d04368fbe026c4 2023-04-12 02:54:00,651 DEBUG [StoreCloser-testSplitTableRegionUnevenDaughter,,1681268036948.e2a9b8fe15a781798c0b49e48ff5aad7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/e2a9b8fe15a781798c0b49e48ff5aad7/cf2/8df43cbe10c74ad8b2c721f3509bb1d1.cc9012ec0b656baa24d04368fbe026c4 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegionUnevenDaughter/e2a9b8fe15a781798c0b49e48ff5aad7/cf2/8df43cbe10c74ad8b2c721f3509bb1d1.cc9012ec0b656baa24d04368fbe026c4 2023-04-12 02:54:00,652 DEBUG [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(130): Registering adapter for the MetricRegistry: RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-04-12 02:54:00,653 INFO [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(134): Registering RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint Metrics about HBase RegionObservers 2023-04-12 02:54:00,653 DEBUG [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(130): Registering adapter for the MetricRegistry: RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro 2023-04-12 02:54:00,653 INFO [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(134): Registering RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro Metrics about HBase RegionObservers 2023-04-12 02:54:00,654 DEBUG [StoreCloser-testSplitTableRegionUnevenDaughter,,1681268036948.e2a9b8fe15a781798c0b49e48ff5aad7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/e2a9b8fe15a781798c0b49e48ff5aad7/cf2/1b7cac025fb940cab9783583b39cef46.cc9012ec0b656baa24d04368fbe026c4 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegionUnevenDaughter/e2a9b8fe15a781798c0b49e48ff5aad7/cf2/1b7cac025fb940cab9783583b39cef46.cc9012ec0b656baa24d04368fbe026c4 2023-04-12 02:54:00,655 DEBUG [StoreCloser-testSplitTableRegionUnevenDaughter,,1681268036948.e2a9b8fe15a781798c0b49e48ff5aad7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/e2a9b8fe15a781798c0b49e48ff5aad7/cf2/213df881d096427c9a5864569db2e1c4.cc9012ec0b656baa24d04368fbe026c4 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegionUnevenDaughter/e2a9b8fe15a781798c0b49e48ff5aad7/cf2/213df881d096427c9a5864569db2e1c4.cc9012ec0b656baa24d04368fbe026c4 2023-04-12 02:54:00,661 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionUnevenDaughter/e2a9b8fe15a781798c0b49e48ff5aad7/recovered.edits/90.seqid, newMaxSeqId=90, maxSeqId=85 2023-04-12 02:54:00,662 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro 2023-04-12 02:54:00,662 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed testSplitTableRegionUnevenDaughter,,1681268036948.e2a9b8fe15a781798c0b49e48ff5aad7. 2023-04-12 02:54:00,662 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for e2a9b8fe15a781798c0b49e48ff5aad7: 2023-04-12 02:54:00,665 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed e2a9b8fe15a781798c0b49e48ff5aad7 2023-04-12 02:54:00,666 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=20 updating hbase:meta row=e2a9b8fe15a781798c0b49e48ff5aad7, regionState=CLOSED 2023-04-12 02:54:00,666 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testSplitTableRegionUnevenDaughter,,1681268036948.e2a9b8fe15a781798c0b49e48ff5aad7.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1681268040666"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681268040666"}]},"ts":"1681268040666"} 2023-04-12 02:54:00,672 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=23, resume processing ppid=20 2023-04-12 02:54:00,672 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=23, ppid=20, state=SUCCESS; CloseRegionProcedure e2a9b8fe15a781798c0b49e48ff5aad7, server=jenkins-hbase4.apache.org,44679,1681268031772 in 375 msec 2023-04-12 02:54:00,676 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=20, resume processing ppid=19 2023-04-12 02:54:00,676 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=20, ppid=19, state=SUCCESS; TransitRegionStateProcedure table=testSplitTableRegionUnevenDaughter, region=e2a9b8fe15a781798c0b49e48ff5aad7, UNASSIGN in 399 msec 2023-04-12 02:54:00,680 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testSplitTableRegionUnevenDaughter","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681268040679"}]},"ts":"1681268040679"} 2023-04-12 02:54:00,682 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testSplitTableRegionUnevenDaughter, state=DISABLED in hbase:meta 2023-04-12 02:54:00,685 INFO [PEWorker-1] procedure.DisableTableProcedure(305): Set testSplitTableRegionUnevenDaughter to state=DISABLED 2023-04-12 02:54:00,690 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter in 442 msec 2023-04-12 02:54:00,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] master.MasterRpcServices(1227): Checking to see if procedure is done pid=19 2023-04-12 02:54:00,878 INFO [Listener at localhost/46667] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testSplitTableRegionUnevenDaughter, procId: 19 completed 2023-04-12 02:54:00,884 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete testSplitTableRegionUnevenDaughter 2023-04-12 02:54:00,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] procedure2.ProcedureExecutor(1029): Stored pid=24, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-12 02:54:00,896 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=24, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-12 02:54:00,899 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=24, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-12 02:54:00,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-12 02:54:00,920 DEBUG [HFileArchiver-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4 2023-04-12 02:54:00,929 DEBUG [HFileArchiver-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitTableRegionUnevenDaughter/e2a9b8fe15a781798c0b49e48ff5aad7 2023-04-12 02:54:00,933 DEBUG [HFileArchiver-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitTableRegionUnevenDaughter/6f55b5d6a460bd8c7569dee22f049104 2023-04-12 02:54:00,937 DEBUG [HFileArchiver-3] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4/cf1, FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4/cf2, FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4/recovered.edits] 2023-04-12 02:54:00,937 DEBUG [HFileArchiver-4] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitTableRegionUnevenDaughter/e2a9b8fe15a781798c0b49e48ff5aad7/cf1, FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitTableRegionUnevenDaughter/e2a9b8fe15a781798c0b49e48ff5aad7/cf2, FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitTableRegionUnevenDaughter/e2a9b8fe15a781798c0b49e48ff5aad7/recovered.edits] 2023-04-12 02:54:00,938 DEBUG [HFileArchiver-5] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitTableRegionUnevenDaughter/6f55b5d6a460bd8c7569dee22f049104/cf1, FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitTableRegionUnevenDaughter/6f55b5d6a460bd8c7569dee22f049104/cf2, FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitTableRegionUnevenDaughter/6f55b5d6a460bd8c7569dee22f049104/recovered.edits] 2023-04-12 02:54:00,946 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4/cf1/07c3d7abf05f43cdbe44fe637da34a7d to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4/cf1/07c3d7abf05f43cdbe44fe637da34a7d 2023-04-12 02:54:00,951 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4/cf1/364127255e644c3d8f0a55272e1ab83a to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4/cf1/364127255e644c3d8f0a55272e1ab83a 2023-04-12 02:54:00,960 DEBUG [HFileArchiver-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitTableRegionUnevenDaughter/e2a9b8fe15a781798c0b49e48ff5aad7/cf1/6bf68177d71a430692424060c2986643 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegionUnevenDaughter/e2a9b8fe15a781798c0b49e48ff5aad7/cf1/6bf68177d71a430692424060c2986643 2023-04-12 02:54:00,960 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4/cf1/5743ff156f3a424f80d4de48581aeeae to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4/cf1/5743ff156f3a424f80d4de48581aeeae 2023-04-12 02:54:00,960 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitTableRegionUnevenDaughter/6f55b5d6a460bd8c7569dee22f049104/cf1/9529f8592dc64944803ca7ce757aed35 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegionUnevenDaughter/6f55b5d6a460bd8c7569dee22f049104/cf1/9529f8592dc64944803ca7ce757aed35 2023-04-12 02:54:00,970 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4/cf1/73d3880f14274287a94d30f14b9cf96e to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4/cf1/73d3880f14274287a94d30f14b9cf96e 2023-04-12 02:54:00,972 DEBUG [HFileArchiver-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitTableRegionUnevenDaughter/e2a9b8fe15a781798c0b49e48ff5aad7/cf2/67d6621e91af49eaab0011c0742d2c28 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegionUnevenDaughter/e2a9b8fe15a781798c0b49e48ff5aad7/cf2/67d6621e91af49eaab0011c0742d2c28 2023-04-12 02:54:00,972 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitTableRegionUnevenDaughter/6f55b5d6a460bd8c7569dee22f049104/cf2/e3c99bab060c48aeaade535a58895af7 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegionUnevenDaughter/6f55b5d6a460bd8c7569dee22f049104/cf2/e3c99bab060c48aeaade535a58895af7 2023-04-12 02:54:00,973 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4/cf1/a32265ffc17446ddb889acfdd1472f9c to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4/cf1/a32265ffc17446ddb889acfdd1472f9c 2023-04-12 02:54:00,979 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitTableRegionUnevenDaughter/6f55b5d6a460bd8c7569dee22f049104/recovered.edits/90.seqid to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegionUnevenDaughter/6f55b5d6a460bd8c7569dee22f049104/recovered.edits/90.seqid 2023-04-12 02:54:00,979 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4/cf1/a859f22056324f46889984d0e730b4bf to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4/cf1/a859f22056324f46889984d0e730b4bf 2023-04-12 02:54:00,981 DEBUG [HFileArchiver-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitTableRegionUnevenDaughter/e2a9b8fe15a781798c0b49e48ff5aad7/recovered.edits/90.seqid to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegionUnevenDaughter/e2a9b8fe15a781798c0b49e48ff5aad7/recovered.edits/90.seqid 2023-04-12 02:54:00,981 DEBUG [HFileArchiver-5] backup.HFileArchiver(596): Deleted hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitTableRegionUnevenDaughter/6f55b5d6a460bd8c7569dee22f049104 2023-04-12 02:54:00,982 DEBUG [HFileArchiver-4] backup.HFileArchiver(596): Deleted hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitTableRegionUnevenDaughter/e2a9b8fe15a781798c0b49e48ff5aad7 2023-04-12 02:54:00,983 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4/cf1/b2cffa1e1e9f40ba81d73ca9452dfeb4 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4/cf1/b2cffa1e1e9f40ba81d73ca9452dfeb4 2023-04-12 02:54:00,988 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4/cf2/18c4c7a41a6d44f9a7eb1d9d77806e17 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4/cf2/18c4c7a41a6d44f9a7eb1d9d77806e17 2023-04-12 02:54:00,990 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4/cf2/1b7cac025fb940cab9783583b39cef46 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4/cf2/1b7cac025fb940cab9783583b39cef46 2023-04-12 02:54:00,992 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4/cf2/213df881d096427c9a5864569db2e1c4 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4/cf2/213df881d096427c9a5864569db2e1c4 2023-04-12 02:54:00,994 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4/cf2/260b76cda6d74251b52568d0821466a2 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4/cf2/260b76cda6d74251b52568d0821466a2 2023-04-12 02:54:00,996 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4/cf2/3310af0091c24e11ae1d57719b30d33b to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4/cf2/3310af0091c24e11ae1d57719b30d33b 2023-04-12 02:54:00,998 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4/cf2/7ed05e97ddbc4e359abe117e7e8b3de9 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4/cf2/7ed05e97ddbc4e359abe117e7e8b3de9 2023-04-12 02:54:01,000 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4/cf2/8df43cbe10c74ad8b2c721f3509bb1d1 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4/cf2/8df43cbe10c74ad8b2c721f3509bb1d1 2023-04-12 02:54:01,004 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4/recovered.edits/85.seqid to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4/recovered.edits/85.seqid 2023-04-12 02:54:01,005 DEBUG [HFileArchiver-3] backup.HFileArchiver(596): Deleted hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitTableRegionUnevenDaughter/cc9012ec0b656baa24d04368fbe026c4 2023-04-12 02:54:01,005 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived testSplitTableRegionUnevenDaughter regions 2023-04-12 02:54:01,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-12 02:54:01,009 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=24, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-12 02:54:01,056 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 3 rows of testSplitTableRegionUnevenDaughter from hbase:meta 2023-04-12 02:54:01,131 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'testSplitTableRegionUnevenDaughter' descriptor. 2023-04-12 02:54:01,134 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=24, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-12 02:54:01,134 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'testSplitTableRegionUnevenDaughter' from region states. 2023-04-12 02:54:01,134 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testSplitTableRegionUnevenDaughter,,1681268034745.cc9012ec0b656baa24d04368fbe026c4.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1681268041134"}]},"ts":"9223372036854775807"} 2023-04-12 02:54:01,134 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testSplitTableRegionUnevenDaughter,,1681268036948.e2a9b8fe15a781798c0b49e48ff5aad7.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1681268041134"}]},"ts":"9223372036854775807"} 2023-04-12 02:54:01,134 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testSplitTableRegionUnevenDaughter,26,1681268036948.6f55b5d6a460bd8c7569dee22f049104.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1681268041134"}]},"ts":"9223372036854775807"} 2023-04-12 02:54:01,143 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 3 regions from META 2023-04-12 02:54:01,143 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => cc9012ec0b656baa24d04368fbe026c4, NAME => 'testSplitTableRegionUnevenDaughter,,1681268034745.cc9012ec0b656baa24d04368fbe026c4.', STARTKEY => '', ENDKEY => ''}, {ENCODED => e2a9b8fe15a781798c0b49e48ff5aad7, NAME => 'testSplitTableRegionUnevenDaughter,,1681268036948.e2a9b8fe15a781798c0b49e48ff5aad7.', STARTKEY => '', ENDKEY => '26'}, {ENCODED => 6f55b5d6a460bd8c7569dee22f049104, NAME => 'testSplitTableRegionUnevenDaughter,26,1681268036948.6f55b5d6a460bd8c7569dee22f049104.', STARTKEY => '26', ENDKEY => ''}] 2023-04-12 02:54:01,143 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'testSplitTableRegionUnevenDaughter' as deleted. 2023-04-12 02:54:01,144 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testSplitTableRegionUnevenDaughter","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1681268041144"}]},"ts":"9223372036854775807"} 2023-04-12 02:54:01,148 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table testSplitTableRegionUnevenDaughter state from META 2023-04-12 02:54:01,151 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=24, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-12 02:54:01,156 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter in 265 msec 2023-04-12 02:54:01,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-12 02:54:01,210 INFO [Listener at localhost/46667] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:testSplitTableRegionUnevenDaughter, procId: 24 completed 2023-04-12 02:54:01,235 INFO [Listener at localhost/46667] hbase.ResourceChecker(175): after: master.assignment.TestSplitTableRegionProcedure#testSplitTableRegionUnevenDaughter Thread=390 (was 359) Potentially hanging thread: HFileArchiver-4 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x6cb27db9-shared-pool-7 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1909933068_15 at /127.0.0.1:46744 [Waiting for operation #8] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1909933068_15 at /127.0.0.1:46690 [Waiting for operation #16] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x6cb27db9-shared-pool-10 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x2d8012cd-shared-pool-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1909933068_15 at /127.0.0.1:37110 [Waiting for operation #2] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1909933068_15 at /127.0.0.1:37036 [Waiting for operation #3] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1909933068_15 at /127.0.0.1:37024 [Waiting for operation #4] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1909933068_15 at /127.0.0.1:36734 [Waiting for operation #32] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x2d8012cd-shared-pool-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x2d8012cd-metaLookup-shared--pool-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1909933068_15 at /127.0.0.1:37064 [Waiting for operation #3] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x6cb27db9-shared-pool-6 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1909933068_15 at /127.0.0.1:46586 [Waiting for operation #9] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RSProcedureDispatcher-pool-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-2 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:883) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1909933068_15 at /127.0.0.1:37042 [Waiting for operation #2] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1909933068_15 at /127.0.0.1:37084 [Waiting for operation #3] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x2d8012cd-metaLookup-shared--pool-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: (jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Timer for 'HBase' metrics system java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1909933068_15 at /127.0.0.1:37108 [Waiting for operation #2] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-5 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x6cb27db9-shared-pool-8 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS:1;jenkins-hbase4:44679-shortCompactions-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.PriorityBlockingQueue.take(PriorityBlockingQueue.java:549) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x2d8012cd-shared-pool-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1909933068_15 at /127.0.0.1:37094 [Waiting for operation #2] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1909933068_15 at /127.0.0.1:46616 [Waiting for operation #7] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x2d8012cd-shared-pool-4 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1909933068_15 at /127.0.0.1:36838 [Waiting for operation #12] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1909933068_15 at /127.0.0.1:37072 [Waiting for operation #2] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x6cb27db9-shared-pool-9 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1909933068_15 at /127.0.0.1:37008 [Waiting for operation #4] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) - Thread LEAK? -, OpenFileDescriptor=844 (was 691) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=376 (was 340) - SystemLoadAverage LEAK? -, ProcessCount=169 (was 169), AvailableMemoryMB=1477 (was 1739) 2023-04-12 02:54:01,267 INFO [Listener at localhost/46667] hbase.ResourceChecker(147): before: master.assignment.TestSplitTableRegionProcedure#testSplitWithoutPONR Thread=390, OpenFileDescriptor=844, MaxFileDescriptor=60000, SystemLoadAverage=376, ProcessCount=169, AvailableMemoryMB=1477 2023-04-12 02:54:01,268 WARN [Listener at localhost/46667] procedure2.ProcedureTestingUtility(193): Set Kill before store update to: false 2023-04-12 02:54:01,278 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/balancer 2023-04-12 02:54:01,278 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] master.MasterRpcServices(492): Client=jenkins//172.31.14.131 set balanceSwitch=false 2023-04-12 02:54:01,281 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(1029): Stored pid=25, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=testSplitWithoutPONR 2023-04-12 02:54:01,284 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=25, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=testSplitWithoutPONR execute state=CREATE_TABLE_PRE_OPERATION 2023-04-12 02:54:01,286 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=25, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=testSplitWithoutPONR execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-12 02:54:01,289 DEBUG [HFileArchiver-6] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d 2023-04-12 02:54:01,290 DEBUG [HFileArchiver-6] backup.HFileArchiver(153): Directory hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d empty. 2023-04-12 02:54:01,290 DEBUG [HFileArchiver-6] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d 2023-04-12 02:54:01,290 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived testSplitWithoutPONR regions 2023-04-12 02:54:01,307 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitWithoutPONR/.tabledesc/.tableinfo.0000000001 2023-04-12 02:54:01,308 INFO [RegionOpenAndInit-testSplitWithoutPONR-pool-0] regionserver.HRegion(7675): creating {ENCODED => 315e13e868a7f68a3ce37802b2f6312d, NAME => 'testSplitWithoutPONR,,1681268041279.315e13e868a7f68a3ce37802b2f6312d.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='testSplitWithoutPONR', {NAME => 'cf1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'cf2', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp 2023-04-12 02:54:01,332 DEBUG [RegionOpenAndInit-testSplitWithoutPONR-pool-0] regionserver.HRegion(866): Instantiated testSplitWithoutPONR,,1681268041279.315e13e868a7f68a3ce37802b2f6312d.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-12 02:54:01,332 DEBUG [RegionOpenAndInit-testSplitWithoutPONR-pool-0] regionserver.HRegion(1604): Closing 315e13e868a7f68a3ce37802b2f6312d, disabling compactions & flushes 2023-04-12 02:54:01,332 INFO [RegionOpenAndInit-testSplitWithoutPONR-pool-0] regionserver.HRegion(1626): Closing region testSplitWithoutPONR,,1681268041279.315e13e868a7f68a3ce37802b2f6312d. 2023-04-12 02:54:01,332 DEBUG [RegionOpenAndInit-testSplitWithoutPONR-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testSplitWithoutPONR,,1681268041279.315e13e868a7f68a3ce37802b2f6312d. 2023-04-12 02:54:01,332 DEBUG [RegionOpenAndInit-testSplitWithoutPONR-pool-0] regionserver.HRegion(1714): Acquired close lock on testSplitWithoutPONR,,1681268041279.315e13e868a7f68a3ce37802b2f6312d. after waiting 0 ms 2023-04-12 02:54:01,332 DEBUG [RegionOpenAndInit-testSplitWithoutPONR-pool-0] regionserver.HRegion(1724): Updates disabled for region testSplitWithoutPONR,,1681268041279.315e13e868a7f68a3ce37802b2f6312d. 2023-04-12 02:54:01,332 INFO [RegionOpenAndInit-testSplitWithoutPONR-pool-0] regionserver.HRegion(1838): Closed testSplitWithoutPONR,,1681268041279.315e13e868a7f68a3ce37802b2f6312d. 2023-04-12 02:54:01,333 DEBUG [RegionOpenAndInit-testSplitWithoutPONR-pool-0] regionserver.HRegion(1558): Region close journal for 315e13e868a7f68a3ce37802b2f6312d: 2023-04-12 02:54:01,337 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=25, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=testSplitWithoutPONR execute state=CREATE_TABLE_ADD_TO_META 2023-04-12 02:54:01,338 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testSplitWithoutPONR,,1681268041279.315e13e868a7f68a3ce37802b2f6312d.","families":{"info":[{"qualifier":"regioninfo","vlen":54,"tag":[],"timestamp":"1681268041338"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681268041338"}]},"ts":"1681268041338"} 2023-04-12 02:54:01,341 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-12 02:54:01,342 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=25, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=testSplitWithoutPONR execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-12 02:54:01,342 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testSplitWithoutPONR","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681268041342"}]},"ts":"1681268041342"} 2023-04-12 02:54:01,346 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testSplitWithoutPONR, state=ENABLING in hbase:meta 2023-04-12 02:54:01,355 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-04-12 02:54:01,357 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-04-12 02:54:01,357 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-04-12 02:54:01,357 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-04-12 02:54:01,357 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-04-12 02:54:01,357 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=26, ppid=25, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSplitWithoutPONR, region=315e13e868a7f68a3ce37802b2f6312d, ASSIGN}] 2023-04-12 02:54:01,359 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=26, ppid=25, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSplitWithoutPONR, region=315e13e868a7f68a3ce37802b2f6312d, ASSIGN 2023-04-12 02:54:01,360 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=26, ppid=25, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testSplitWithoutPONR, region=315e13e868a7f68a3ce37802b2f6312d, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,34579,1681268031719; forceNewPlan=false, retain=false 2023-04-12 02:54:01,510 INFO [jenkins-hbase4:40281] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-04-12 02:54:01,512 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=26 updating hbase:meta row=315e13e868a7f68a3ce37802b2f6312d, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:01,512 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitWithoutPONR,,1681268041279.315e13e868a7f68a3ce37802b2f6312d.","families":{"info":[{"qualifier":"regioninfo","vlen":54,"tag":[],"timestamp":"1681268041512"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681268041512"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681268041512"}]},"ts":"1681268041512"} 2023-04-12 02:54:01,515 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=27, ppid=26, state=RUNNABLE; OpenRegionProcedure 315e13e868a7f68a3ce37802b2f6312d, server=jenkins-hbase4.apache.org,34579,1681268031719}] 2023-04-12 02:54:01,669 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:01,670 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-04-12 02:54:01,682 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:49288, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-04-12 02:54:01,699 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testSplitWithoutPONR,,1681268041279.315e13e868a7f68a3ce37802b2f6312d. 2023-04-12 02:54:01,700 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 315e13e868a7f68a3ce37802b2f6312d, NAME => 'testSplitWithoutPONR,,1681268041279.315e13e868a7f68a3ce37802b2f6312d.', STARTKEY => '', ENDKEY => ''} 2023-04-12 02:54:01,700 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(174): System coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro loaded, priority=536870911. 2023-04-12 02:54:01,700 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testSplitWithoutPONR 315e13e868a7f68a3ce37802b2f6312d 2023-04-12 02:54:01,700 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated testSplitWithoutPONR,,1681268041279.315e13e868a7f68a3ce37802b2f6312d.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-12 02:54:01,700 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 315e13e868a7f68a3ce37802b2f6312d 2023-04-12 02:54:01,700 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 315e13e868a7f68a3ce37802b2f6312d 2023-04-12 02:54:01,711 INFO [StoreOpener-315e13e868a7f68a3ce37802b2f6312d-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf1 of region 315e13e868a7f68a3ce37802b2f6312d 2023-04-12 02:54:01,715 DEBUG [StoreOpener-315e13e868a7f68a3ce37802b2f6312d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf1 2023-04-12 02:54:01,715 DEBUG [StoreOpener-315e13e868a7f68a3ce37802b2f6312d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf1 2023-04-12 02:54:01,716 INFO [StoreOpener-315e13e868a7f68a3ce37802b2f6312d-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 315e13e868a7f68a3ce37802b2f6312d columnFamilyName cf1 2023-04-12 02:54:01,726 INFO [StoreOpener-315e13e868a7f68a3ce37802b2f6312d-1] regionserver.HStore(310): Store=315e13e868a7f68a3ce37802b2f6312d/cf1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-12 02:54:01,726 INFO [StoreOpener-315e13e868a7f68a3ce37802b2f6312d-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf2 of region 315e13e868a7f68a3ce37802b2f6312d 2023-04-12 02:54:01,729 DEBUG [StoreOpener-315e13e868a7f68a3ce37802b2f6312d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf2 2023-04-12 02:54:01,729 DEBUG [StoreOpener-315e13e868a7f68a3ce37802b2f6312d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf2 2023-04-12 02:54:01,731 INFO [StoreOpener-315e13e868a7f68a3ce37802b2f6312d-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 315e13e868a7f68a3ce37802b2f6312d columnFamilyName cf2 2023-04-12 02:54:01,732 INFO [StoreOpener-315e13e868a7f68a3ce37802b2f6312d-1] regionserver.HStore(310): Store=315e13e868a7f68a3ce37802b2f6312d/cf2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-12 02:54:01,733 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d 2023-04-12 02:54:01,734 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d 2023-04-12 02:54:01,741 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testSplitWithoutPONR descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-04-12 02:54:01,752 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 315e13e868a7f68a3ce37802b2f6312d 2023-04-12 02:54:01,771 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-12 02:54:01,772 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 315e13e868a7f68a3ce37802b2f6312d; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=72533692, jitterRate=0.0808362364768982}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-04-12 02:54:01,772 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 315e13e868a7f68a3ce37802b2f6312d: 2023-04-12 02:54:01,773 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testSplitWithoutPONR,,1681268041279.315e13e868a7f68a3ce37802b2f6312d., pid=27, masterSystemTime=1681268041669 2023-04-12 02:54:01,780 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testSplitWithoutPONR,,1681268041279.315e13e868a7f68a3ce37802b2f6312d. 2023-04-12 02:54:01,781 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=26 updating hbase:meta row=315e13e868a7f68a3ce37802b2f6312d, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:01,781 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testSplitWithoutPONR,,1681268041279.315e13e868a7f68a3ce37802b2f6312d.","families":{"info":[{"qualifier":"regioninfo","vlen":54,"tag":[],"timestamp":"1681268041781"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1681268041781"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681268041781"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681268041781"}]},"ts":"1681268041781"} 2023-04-12 02:54:01,782 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testSplitWithoutPONR,,1681268041279.315e13e868a7f68a3ce37802b2f6312d. 2023-04-12 02:54:01,792 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=27, resume processing ppid=26 2023-04-12 02:54:01,792 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=27, ppid=26, state=SUCCESS; OpenRegionProcedure 315e13e868a7f68a3ce37802b2f6312d, server=jenkins-hbase4.apache.org,34579,1681268031719 in 272 msec 2023-04-12 02:54:01,797 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=26, resume processing ppid=25 2023-04-12 02:54:01,797 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=26, ppid=25, state=SUCCESS; TransitRegionStateProcedure table=testSplitWithoutPONR, region=315e13e868a7f68a3ce37802b2f6312d, ASSIGN in 435 msec 2023-04-12 02:54:01,799 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=25, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=testSplitWithoutPONR execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-12 02:54:01,799 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testSplitWithoutPONR","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681268041799"}]},"ts":"1681268041799"} 2023-04-12 02:54:01,802 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testSplitWithoutPONR, state=ENABLED in hbase:meta 2023-04-12 02:54:01,810 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=25, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=testSplitWithoutPONR execute state=CREATE_TABLE_POST_OPERATION 2023-04-12 02:54:01,815 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR in 531 msec 2023-04-12 02:54:02,041 DEBUG [Listener at localhost/46667] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-12 02:54:02,049 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:49302, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-12 02:54:02,067 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 02:54:02,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] procedure.ProcedureCoordinator(165): Submitting procedure testSplitWithoutPONR 2023-04-12 02:54:02,079 INFO [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testSplitWithoutPONR' 2023-04-12 02:54:02,079 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:54:02,080 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testSplitWithoutPONR' starting 'acquire' 2023-04-12 02:54:02,080 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testSplitWithoutPONR', kicking off acquire phase on members. 2023-04-12 02:54:02,081 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-12 02:54:02,081 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-04-12 02:54:02,084 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:54:02,084 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:02,084 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:54:02,084 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:54:02,084 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:54:02,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:54:02,084 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:54:02,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:54:02,085 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:02,085 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 02:54:02,084 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:54:02,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:54:02,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-04-12 02:54:02,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-04-12 02:54:02,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-04-12 02:54:02,085 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-12 02:54:02,086 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-12 02:54:02,086 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-12 02:54:02,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 02:54:02,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 02:54:02,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-04-12 02:54:02,086 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitWithoutPONR 2023-04-12 02:54:02,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-04-12 02:54:02,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 02:54:02,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-04-12 02:54:02,087 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitWithoutPONR 2023-04-12 02:54:02,086 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitWithoutPONR 2023-04-12 02:54:02,099 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitWithoutPONR 2023-04-12 02:54:02,099 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitWithoutPONR' with timeout 60000ms 2023-04-12 02:54:02,099 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:54:02,108 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitWithoutPONR 2023-04-12 02:54:02,109 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitWithoutPONR 2023-04-12 02:54:02,109 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitWithoutPONR' starting 'acquire' stage 2023-04-12 02:54:02,109 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitWithoutPONR' with timeout 60000ms 2023-04-12 02:54:02,109 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitWithoutPONR' locally acquired 2023-04-12 02:54:02,109 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitWithoutPONR' with timeout 60000ms 2023-04-12 02:54:02,109 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43091,1681268031804' joining acquired barrier for procedure (testSplitWithoutPONR) in zk 2023-04-12 02:54:02,109 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:54:02,109 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:54:02,110 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitWithoutPONR' starting 'acquire' stage 2023-04-12 02:54:02,111 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitWithoutPONR' starting 'acquire' stage 2023-04-12 02:54:02,111 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitWithoutPONR' locally acquired 2023-04-12 02:54:02,111 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44679,1681268031772' joining acquired barrier for procedure (testSplitWithoutPONR) in zk 2023-04-12 02:54:02,111 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 02:54:02,111 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 02:54:02,111 DEBUG [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testSplitWithoutPONR,,1681268041279.315e13e868a7f68a3ce37802b2f6312d. 2023-04-12 02:54:02,111 DEBUG [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testSplitWithoutPONR,,1681268041279.315e13e868a7f68a3ce37802b2f6312d. started... 2023-04-12 02:54:02,112 INFO [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 315e13e868a7f68a3ce37802b2f6312d 2/2 column families, dataSize=120 B heapSize=928 B 2023-04-12 02:54:02,113 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-12 02:54:02,114 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-12 02:54:02,114 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-12 02:54:02,114 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitWithoutPONR' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 02:54:02,115 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-12 02:54:02,115 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitWithoutPONR' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 02:54:02,160 INFO [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=60 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/.tmp/cf1/1cabdcde78c14f6a8ec70790c1ae4cc4 2023-04-12 02:54:02,218 INFO [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=60 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/.tmp/cf2/0b28a4ef23e442f38cfdf4760e6f9480 2023-04-12 02:54:02,226 DEBUG [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/.tmp/cf1/1cabdcde78c14f6a8ec70790c1ae4cc4 as hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf1/1cabdcde78c14f6a8ec70790c1ae4cc4 2023-04-12 02:54:02,239 INFO [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf1/1cabdcde78c14f6a8ec70790c1ae4cc4, entries=2, sequenceid=6, filesize=4.8 K 2023-04-12 02:54:02,240 DEBUG [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/.tmp/cf2/0b28a4ef23e442f38cfdf4760e6f9480 as hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf2/0b28a4ef23e442f38cfdf4760e6f9480 2023-04-12 02:54:02,250 INFO [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf2/0b28a4ef23e442f38cfdf4760e6f9480, entries=2, sequenceid=6, filesize=4.8 K 2023-04-12 02:54:02,255 INFO [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~120 B/120, heapSize ~896 B/896, currentSize=0 B/0 for 315e13e868a7f68a3ce37802b2f6312d in 143ms, sequenceid=6, compaction requested=false 2023-04-12 02:54:02,256 DEBUG [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'testSplitWithoutPONR' 2023-04-12 02:54:02,257 DEBUG [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 315e13e868a7f68a3ce37802b2f6312d: 2023-04-12 02:54:02,257 DEBUG [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testSplitWithoutPONR,,1681268041279.315e13e868a7f68a3ce37802b2f6312d. 2023-04-12 02:54:02,257 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 02:54:02,257 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 02:54:02,257 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:02,257 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitWithoutPONR' locally acquired 2023-04-12 02:54:02,257 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34579,1681268031719' joining acquired barrier for procedure (testSplitWithoutPONR) in zk 2023-04-12 02:54:02,259 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-12 02:54:02,259 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:02,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:02,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 02:54:02,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 02:54:02,260 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-12 02:54:02,260 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitWithoutPONR' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 02:54:02,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 02:54:02,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 02:54:02,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitWithoutPONR 2023-04-12 02:54:02,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:02,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:02,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:02,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 02:54:02,266 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34579,1681268031719' joining acquired barrier for procedure 'testSplitWithoutPONR' on coordinator 2023-04-12 02:54:02,266 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6428d2f7[Count = 0] remaining members to acquire global barrier 2023-04-12 02:54:02,266 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testSplitWithoutPONR' starting 'in-barrier' execution. 2023-04-12 02:54:02,266 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-12 02:54:02,268 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-12 02:54:02,268 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-12 02:54:02,268 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-12 02:54:02,268 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-12 02:54:02,268 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:02,268 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 02:54:02,268 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-12 02:54:02,269 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-12 02:54:02,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-12 02:54:02,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-12 02:54:02,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-12 02:54:02,269 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitWithoutPONR' received 'reached' from coordinator. 2023-04-12 02:54:02,269 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitWithoutPONR' received 'reached' from coordinator. 2023-04-12 02:54:02,269 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitWithoutPONR' locally completed 2023-04-12 02:54:02,269 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitWithoutPONR' locally completed 2023-04-12 02:54:02,269 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitWithoutPONR' completed for member 'jenkins-hbase4.apache.org,34579,1681268031719' in zk 2023-04-12 02:54:02,269 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitWithoutPONR' received 'reached' from coordinator. 2023-04-12 02:54:02,269 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitWithoutPONR' locally completed 2023-04-12 02:54:02,269 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitWithoutPONR' completed for member 'jenkins-hbase4.apache.org,43091,1681268031804' in zk 2023-04-12 02:54:02,269 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitWithoutPONR' completed for member 'jenkins-hbase4.apache.org,44679,1681268031772' in zk 2023-04-12 02:54:02,272 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:02,272 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitWithoutPONR' has notified controller of completion 2023-04-12 02:54:02,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:02,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 02:54:02,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 02:54:02,272 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:54:02,273 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitWithoutPONR' completed. 2023-04-12 02:54:02,274 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitWithoutPONR' has notified controller of completion 2023-04-12 02:54:02,275 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitWithoutPONR' has notified controller of completion 2023-04-12 02:54:02,275 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:54:02,275 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:54:02,275 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitWithoutPONR' completed. 2023-04-12 02:54:02,275 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitWithoutPONR' completed. 2023-04-12 02:54:02,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 02:54:02,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 02:54:02,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitWithoutPONR 2023-04-12 02:54:02,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:02,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:02,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:02,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 02:54:02,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitWithoutPONR 2023-04-12 02:54:02,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:02,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:02,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:02,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testSplitWithoutPONR' member 'jenkins-hbase4.apache.org,34579,1681268031719': 2023-04-12 02:54:02,280 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34579,1681268031719' released barrier for procedure'testSplitWithoutPONR', counting down latch. Waiting for 0 more 2023-04-12 02:54:02,281 INFO [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testSplitWithoutPONR' execution completed 2023-04-12 02:54:02,281 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 02:54:02,281 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 02:54:02,281 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testSplitWithoutPONR 2023-04-12 02:54:02,281 INFO [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testSplitWithoutPONRincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 02:54:02,283 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-12 02:54:02,283 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-12 02:54:02,283 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-12 02:54:02,283 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-12 02:54:02,283 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-12 02:54:02,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-12 02:54:02,283 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:54:02,283 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-12 02:54:02,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-12 02:54:02,283 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:54:02,283 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:54:02,283 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-12 02:54:02,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-12 02:54:02,284 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:54:02,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:54:02,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-12 02:54:02,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 02:54:02,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 02:54:02,284 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:54:02,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:54:02,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-12 02:54:02,285 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:54:02,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:54:02,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-12 02:54:02,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 02:54:02,286 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:02,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-12 02:54:02,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitWithoutPONR 2023-04-12 02:54:02,287 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:02,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 02:54:02,288 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:02,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitWithoutPONR 2023-04-12 02:54:02,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:02,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:02,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:02,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 02:54:02,295 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:02,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitWithoutPONR 2023-04-12 02:54:02,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:02,296 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:02,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:02,297 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:02,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:02,302 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:54:02,302 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:54:02,302 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:54:02,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:54:02,302 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:54:02,302 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:02,302 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:54:02,302 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:54:02,302 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:54:02,304 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:54:02,302 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testSplitWithoutPONR' 2023-04-12 02:54:02,304 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 02:54:02,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:54:02,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:54:02,302 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:54:02,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:54:02,302 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-04-12 02:54:02,304 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:02,304 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:02,305 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-04-12 02:54:02,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:54:02,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:54:02,305 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:02,305 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-12 02:54:02,305 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:02,305 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:02,305 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-12 02:54:02,305 DEBUG [Listener at localhost/46667] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testSplitWithoutPONR'' to complete. (max 20000 ms per retry) 2023-04-12 02:54:02,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:54:02,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:54:02,305 DEBUG [Listener at localhost/46667] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 02:54:02,305 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-12 02:54:02,307 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:54:02,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:54:02,406 DEBUG [Listener at localhost/46667] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 02:54:02,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 02:54:02,440 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 02:54:02,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] procedure.ProcedureCoordinator(143): Procedure testSplitWithoutPONR was in running list but was completed. Accepting new attempt. 2023-04-12 02:54:02,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] procedure.ProcedureCoordinator(165): Submitting procedure testSplitWithoutPONR 2023-04-12 02:54:02,449 INFO [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testSplitWithoutPONR' 2023-04-12 02:54:02,449 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:54:02,453 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testSplitWithoutPONR' starting 'acquire' 2023-04-12 02:54:02,454 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testSplitWithoutPONR', kicking off acquire phase on members. 2023-04-12 02:54:02,454 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-12 02:54:02,454 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-04-12 02:54:02,456 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:54:02,456 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:54:02,456 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:54:02,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:54:02,456 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:54:02,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:54:02,456 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:54:02,456 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:02,457 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:54:02,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:54:02,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-04-12 02:54:02,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-04-12 02:54:02,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-04-12 02:54:02,457 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:02,457 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 02:54:02,457 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-12 02:54:02,458 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-12 02:54:02,458 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-12 02:54:02,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 02:54:02,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-04-12 02:54:02,458 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitWithoutPONR 2023-04-12 02:54:02,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 02:54:02,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-04-12 02:54:02,459 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitWithoutPONR 2023-04-12 02:54:02,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 02:54:02,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-04-12 02:54:02,459 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitWithoutPONR 2023-04-12 02:54:02,461 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitWithoutPONR is still present, removing 2023-04-12 02:54:02,461 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitWithoutPONR 2023-04-12 02:54:02,461 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitWithoutPONR' with timeout 60000ms 2023-04-12 02:54:02,461 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:54:02,463 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitWithoutPONR is still present, removing 2023-04-12 02:54:02,464 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitWithoutPONR is still present, removing 2023-04-12 02:54:02,464 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitWithoutPONR' starting 'acquire' stage 2023-04-12 02:54:02,464 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitWithoutPONR 2023-04-12 02:54:02,464 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitWithoutPONR 2023-04-12 02:54:02,464 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitWithoutPONR' with timeout 60000ms 2023-04-12 02:54:02,464 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitWithoutPONR' with timeout 60000ms 2023-04-12 02:54:02,464 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:54:02,464 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:54:02,465 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitWithoutPONR' starting 'acquire' stage 2023-04-12 02:54:02,465 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitWithoutPONR' starting 'acquire' stage 2023-04-12 02:54:02,464 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitWithoutPONR' locally acquired 2023-04-12 02:54:02,465 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitWithoutPONR' locally acquired 2023-04-12 02:54:02,465 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43091,1681268031804' joining acquired barrier for procedure (testSplitWithoutPONR) in zk 2023-04-12 02:54:02,466 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 02:54:02,466 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 02:54:02,466 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44679,1681268031772' joining acquired barrier for procedure (testSplitWithoutPONR) in zk 2023-04-12 02:54:02,466 DEBUG [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testSplitWithoutPONR,,1681268041279.315e13e868a7f68a3ce37802b2f6312d. 2023-04-12 02:54:02,466 DEBUG [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testSplitWithoutPONR,,1681268041279.315e13e868a7f68a3ce37802b2f6312d. started... 2023-04-12 02:54:02,466 INFO [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 315e13e868a7f68a3ce37802b2f6312d 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-04-12 02:54:02,468 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-12 02:54:02,468 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-12 02:54:02,468 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-12 02:54:02,468 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitWithoutPONR' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 02:54:02,472 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-12 02:54:02,472 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitWithoutPONR' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 02:54:02,517 INFO [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=19 (bloomFilter=true), to=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/.tmp/cf1/2fd181776c3749c18358aebdfc597014 2023-04-12 02:54:02,544 INFO [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=19 (bloomFilter=true), to=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/.tmp/cf2/2540a68ffbdf4440b5ff30ca6a36fada 2023-04-12 02:54:02,551 DEBUG [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/.tmp/cf1/2fd181776c3749c18358aebdfc597014 as hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf1/2fd181776c3749c18358aebdfc597014 2023-04-12 02:54:02,559 INFO [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf1/2fd181776c3749c18358aebdfc597014, entries=10, sequenceid=19, filesize=5.0 K 2023-04-12 02:54:02,560 DEBUG [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/.tmp/cf2/2540a68ffbdf4440b5ff30ca6a36fada as hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf2/2540a68ffbdf4440b5ff30ca6a36fada 2023-04-12 02:54:02,569 INFO [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf2/2540a68ffbdf4440b5ff30ca6a36fada, entries=10, sequenceid=19, filesize=5.0 K 2023-04-12 02:54:02,571 INFO [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for 315e13e868a7f68a3ce37802b2f6312d in 105ms, sequenceid=19, compaction requested=false 2023-04-12 02:54:02,571 DEBUG [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 315e13e868a7f68a3ce37802b2f6312d: 2023-04-12 02:54:02,571 DEBUG [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testSplitWithoutPONR,,1681268041279.315e13e868a7f68a3ce37802b2f6312d. 2023-04-12 02:54:02,572 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 02:54:02,572 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 02:54:02,572 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:02,572 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitWithoutPONR' locally acquired 2023-04-12 02:54:02,572 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34579,1681268031719' joining acquired barrier for procedure (testSplitWithoutPONR) in zk 2023-04-12 02:54:02,574 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:02,574 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-12 02:54:02,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:02,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 02:54:02,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 02:54:02,574 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-12 02:54:02,574 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitWithoutPONR' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 02:54:02,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 02:54:02,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 02:54:02,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitWithoutPONR 2023-04-12 02:54:02,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:02,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:02,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:02,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 02:54:02,588 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34579,1681268031719' joining acquired barrier for procedure 'testSplitWithoutPONR' on coordinator 2023-04-12 02:54:02,588 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1d65e7fd[Count = 0] remaining members to acquire global barrier 2023-04-12 02:54:02,588 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testSplitWithoutPONR' starting 'in-barrier' execution. 2023-04-12 02:54:02,588 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-12 02:54:02,590 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-12 02:54:02,590 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-12 02:54:02,590 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-12 02:54:02,590 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-12 02:54:02,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-12 02:54:02,590 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-12 02:54:02,591 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitWithoutPONR' received 'reached' from coordinator. 2023-04-12 02:54:02,591 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-12 02:54:02,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-12 02:54:02,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-12 02:54:02,591 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:02,591 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 02:54:02,591 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitWithoutPONR' received 'reached' from coordinator. 2023-04-12 02:54:02,591 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitWithoutPONR' locally completed 2023-04-12 02:54:02,591 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitWithoutPONR' locally completed 2023-04-12 02:54:02,591 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitWithoutPONR' received 'reached' from coordinator. 2023-04-12 02:54:02,591 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitWithoutPONR' completed for member 'jenkins-hbase4.apache.org,44679,1681268031772' in zk 2023-04-12 02:54:02,591 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitWithoutPONR' completed for member 'jenkins-hbase4.apache.org,34579,1681268031719' in zk 2023-04-12 02:54:02,591 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitWithoutPONR' locally completed 2023-04-12 02:54:02,591 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitWithoutPONR' completed for member 'jenkins-hbase4.apache.org,43091,1681268031804' in zk 2023-04-12 02:54:02,596 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitWithoutPONR' has notified controller of completion 2023-04-12 02:54:02,596 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:54:02,596 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitWithoutPONR' completed. 2023-04-12 02:54:02,597 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitWithoutPONR' has notified controller of completion 2023-04-12 02:54:02,598 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:02,598 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:54:02,598 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitWithoutPONR' completed. 2023-04-12 02:54:02,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:02,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 02:54:02,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 02:54:02,599 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitWithoutPONR' has notified controller of completion 2023-04-12 02:54:02,599 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:54:02,599 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitWithoutPONR' completed. 2023-04-12 02:54:02,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 02:54:02,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 02:54:02,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitWithoutPONR 2023-04-12 02:54:02,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:02,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:02,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:02,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 02:54:02,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitWithoutPONR 2023-04-12 02:54:02,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:02,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:02,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:02,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testSplitWithoutPONR' member 'jenkins-hbase4.apache.org,34579,1681268031719': 2023-04-12 02:54:02,608 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34579,1681268031719' released barrier for procedure'testSplitWithoutPONR', counting down latch. Waiting for 0 more 2023-04-12 02:54:02,608 INFO [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testSplitWithoutPONR' execution completed 2023-04-12 02:54:02,608 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 02:54:02,608 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 02:54:02,608 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testSplitWithoutPONR 2023-04-12 02:54:02,608 INFO [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testSplitWithoutPONRincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 02:54:02,611 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-12 02:54:02,611 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-12 02:54:02,612 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:54:02,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-12 02:54:02,611 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-12 02:54:02,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-12 02:54:02,612 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-12 02:54:02,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-12 02:54:02,611 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:54:02,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-12 02:54:02,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-12 02:54:02,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-12 02:54:02,612 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:54:02,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-12 02:54:02,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 02:54:02,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 02:54:02,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:54:02,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:54:02,613 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:54:02,613 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:54:02,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:54:02,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:54:02,613 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:02,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-12 02:54:02,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 02:54:02,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-12 02:54:02,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-12 02:54:02,614 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:02,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitWithoutPONR 2023-04-12 02:54:02,615 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:02,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 02:54:02,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitWithoutPONR 2023-04-12 02:54:02,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:02,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:02,617 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:02,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:02,619 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:02,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 02:54:02,619 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:02,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitWithoutPONR 2023-04-12 02:54:02,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:02,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:02,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:02,624 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:54:02,624 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:54:02,624 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:54:02,624 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:54:02,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:54:02,624 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:54:02,624 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:54:02,624 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:54:02,624 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:54:02,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:54:02,625 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:54:02,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:54:02,625 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:54:02,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:54:02,625 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:54:02,625 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:02,625 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:54:02,625 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-04-12 02:54:02,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:54:02,626 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:02,626 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:02,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:54:02,626 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-04-12 02:54:02,626 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:02,626 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-12 02:54:02,626 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:02,626 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:02,626 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-12 02:54:02,626 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testSplitWithoutPONR' 2023-04-12 02:54:02,626 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 02:54:02,626 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:54:02,626 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-12 02:54:02,627 DEBUG [Listener at localhost/46667] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testSplitWithoutPONR'' to complete. (max 20000 ms per retry) 2023-04-12 02:54:02,627 DEBUG [Listener at localhost/46667] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 02:54:02,727 DEBUG [Listener at localhost/46667] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 02:54:02,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 02:54:02,758 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 02:54:02,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] procedure.ProcedureCoordinator(143): Procedure testSplitWithoutPONR was in running list but was completed. Accepting new attempt. 2023-04-12 02:54:02,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] procedure.ProcedureCoordinator(165): Submitting procedure testSplitWithoutPONR 2023-04-12 02:54:02,760 INFO [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testSplitWithoutPONR' 2023-04-12 02:54:02,760 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:54:02,761 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testSplitWithoutPONR' starting 'acquire' 2023-04-12 02:54:02,761 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testSplitWithoutPONR', kicking off acquire phase on members. 2023-04-12 02:54:02,762 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-12 02:54:02,762 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-04-12 02:54:02,764 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:54:02,764 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:54:02,764 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:54:02,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:54:02,764 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:54:02,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:54:02,764 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:54:02,764 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:54:02,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:54:02,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-04-12 02:54:02,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-04-12 02:54:02,765 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:02,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-04-12 02:54:02,765 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-12 02:54:02,765 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-12 02:54:02,766 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-12 02:54:02,766 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:02,766 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 02:54:02,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 02:54:02,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-04-12 02:54:02,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 02:54:02,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-04-12 02:54:02,766 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitWithoutPONR 2023-04-12 02:54:02,766 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitWithoutPONR 2023-04-12 02:54:02,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 02:54:02,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-04-12 02:54:02,767 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitWithoutPONR 2023-04-12 02:54:02,767 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitWithoutPONR is still present, removing 2023-04-12 02:54:02,767 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitWithoutPONR 2023-04-12 02:54:02,767 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitWithoutPONR is still present, removing 2023-04-12 02:54:02,767 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitWithoutPONR' with timeout 60000ms 2023-04-12 02:54:02,767 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitWithoutPONR is still present, removing 2023-04-12 02:54:02,767 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitWithoutPONR 2023-04-12 02:54:02,767 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitWithoutPONR 2023-04-12 02:54:02,767 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:54:02,767 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitWithoutPONR' with timeout 60000ms 2023-04-12 02:54:02,767 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitWithoutPONR' with timeout 60000ms 2023-04-12 02:54:02,768 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:54:02,768 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitWithoutPONR' starting 'acquire' stage 2023-04-12 02:54:02,768 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:54:02,778 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitWithoutPONR' starting 'acquire' stage 2023-04-12 02:54:02,778 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitWithoutPONR' locally acquired 2023-04-12 02:54:02,778 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44679,1681268031772' joining acquired barrier for procedure (testSplitWithoutPONR) in zk 2023-04-12 02:54:02,778 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitWithoutPONR' locally acquired 2023-04-12 02:54:02,779 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43091,1681268031804' joining acquired barrier for procedure (testSplitWithoutPONR) in zk 2023-04-12 02:54:02,780 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitWithoutPONR' starting 'acquire' stage 2023-04-12 02:54:02,788 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 02:54:02,788 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 02:54:02,788 DEBUG [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testSplitWithoutPONR,,1681268041279.315e13e868a7f68a3ce37802b2f6312d. 2023-04-12 02:54:02,788 DEBUG [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testSplitWithoutPONR,,1681268041279.315e13e868a7f68a3ce37802b2f6312d. started... 2023-04-12 02:54:02,788 INFO [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 315e13e868a7f68a3ce37802b2f6312d 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-04-12 02:54:02,789 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-12 02:54:02,789 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-12 02:54:02,792 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-12 02:54:02,792 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-12 02:54:02,793 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitWithoutPONR' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 02:54:02,792 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitWithoutPONR' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 02:54:02,817 INFO [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=32 (bloomFilter=true), to=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/.tmp/cf1/55bdd10dae634a008e897b8e86048f3c 2023-04-12 02:54:02,868 INFO [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=32 (bloomFilter=true), to=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/.tmp/cf2/babf83b74cec452290ff19cd45eb4713 2023-04-12 02:54:02,876 DEBUG [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/.tmp/cf1/55bdd10dae634a008e897b8e86048f3c as hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf1/55bdd10dae634a008e897b8e86048f3c 2023-04-12 02:54:02,886 INFO [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf1/55bdd10dae634a008e897b8e86048f3c, entries=10, sequenceid=32, filesize=5.0 K 2023-04-12 02:54:02,888 DEBUG [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/.tmp/cf2/babf83b74cec452290ff19cd45eb4713 as hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf2/babf83b74cec452290ff19cd45eb4713 2023-04-12 02:54:02,901 INFO [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf2/babf83b74cec452290ff19cd45eb4713, entries=10, sequenceid=32, filesize=5.0 K 2023-04-12 02:54:02,903 INFO [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for 315e13e868a7f68a3ce37802b2f6312d in 115ms, sequenceid=32, compaction requested=true 2023-04-12 02:54:02,903 DEBUG [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 315e13e868a7f68a3ce37802b2f6312d: 2023-04-12 02:54:02,903 DEBUG [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testSplitWithoutPONR,,1681268041279.315e13e868a7f68a3ce37802b2f6312d. 2023-04-12 02:54:02,903 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 02:54:02,903 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 02:54:02,903 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:02,903 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitWithoutPONR' locally acquired 2023-04-12 02:54:02,903 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34579,1681268031719' joining acquired barrier for procedure (testSplitWithoutPONR) in zk 2023-04-12 02:54:02,907 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-12 02:54:02,907 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:02,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:02,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 02:54:02,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 02:54:02,907 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-12 02:54:02,907 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitWithoutPONR' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 02:54:02,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 02:54:02,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 02:54:02,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitWithoutPONR 2023-04-12 02:54:02,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:02,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:02,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:02,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 02:54:02,911 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34579,1681268031719' joining acquired barrier for procedure 'testSplitWithoutPONR' on coordinator 2023-04-12 02:54:02,912 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1cbbed55[Count = 0] remaining members to acquire global barrier 2023-04-12 02:54:02,912 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testSplitWithoutPONR' starting 'in-barrier' execution. 2023-04-12 02:54:02,912 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-12 02:54:02,913 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-12 02:54:02,913 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-12 02:54:02,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-12 02:54:02,913 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-12 02:54:02,913 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-12 02:54:02,913 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-12 02:54:02,913 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitWithoutPONR' received 'reached' from coordinator. 2023-04-12 02:54:02,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-12 02:54:02,914 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-12 02:54:02,914 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitWithoutPONR' received 'reached' from coordinator. 2023-04-12 02:54:02,914 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitWithoutPONR' locally completed 2023-04-12 02:54:02,914 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitWithoutPONR' completed for member 'jenkins-hbase4.apache.org,43091,1681268031804' in zk 2023-04-12 02:54:02,914 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitWithoutPONR' locally completed 2023-04-12 02:54:02,914 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitWithoutPONR' completed for member 'jenkins-hbase4.apache.org,34579,1681268031719' in zk 2023-04-12 02:54:02,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-12 02:54:02,914 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitWithoutPONR' received 'reached' from coordinator. 2023-04-12 02:54:02,914 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitWithoutPONR' locally completed 2023-04-12 02:54:02,914 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitWithoutPONR' completed for member 'jenkins-hbase4.apache.org,44679,1681268031772' in zk 2023-04-12 02:54:02,915 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:02,915 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 02:54:02,919 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitWithoutPONR' has notified controller of completion 2023-04-12 02:54:02,919 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:54:02,919 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitWithoutPONR' completed. 2023-04-12 02:54:02,921 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:02,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:02,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 02:54:02,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 02:54:02,921 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitWithoutPONR' has notified controller of completion 2023-04-12 02:54:02,921 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitWithoutPONR' has notified controller of completion 2023-04-12 02:54:02,921 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:54:02,921 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitWithoutPONR' completed. 2023-04-12 02:54:02,921 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:54:02,921 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitWithoutPONR' completed. 2023-04-12 02:54:02,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 02:54:02,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 02:54:02,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitWithoutPONR 2023-04-12 02:54:02,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:02,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:02,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:02,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 02:54:02,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitWithoutPONR 2023-04-12 02:54:02,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:02,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:02,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:02,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testSplitWithoutPONR' member 'jenkins-hbase4.apache.org,34579,1681268031719': 2023-04-12 02:54:02,930 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34579,1681268031719' released barrier for procedure'testSplitWithoutPONR', counting down latch. Waiting for 0 more 2023-04-12 02:54:02,930 INFO [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testSplitWithoutPONR' execution completed 2023-04-12 02:54:02,930 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 02:54:02,930 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 02:54:02,930 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testSplitWithoutPONR 2023-04-12 02:54:02,930 INFO [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testSplitWithoutPONRincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 02:54:02,937 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-12 02:54:02,937 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-12 02:54:02,937 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-12 02:54:02,937 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-12 02:54:02,937 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-12 02:54:02,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-12 02:54:02,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-12 02:54:02,937 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:54:02,937 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:54:02,937 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-12 02:54:02,938 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:54:02,938 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-12 02:54:02,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-12 02:54:02,938 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:54:02,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:54:02,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-12 02:54:02,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 02:54:02,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 02:54:02,938 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:54:02,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:54:02,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-12 02:54:02,939 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:54:02,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:54:02,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-12 02:54:02,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 02:54:02,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-12 02:54:02,939 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:02,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitWithoutPONR 2023-04-12 02:54:02,940 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:02,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 02:54:02,945 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:02,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitWithoutPONR 2023-04-12 02:54:02,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:02,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:02,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:02,959 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:02,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 02:54:02,959 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:02,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitWithoutPONR 2023-04-12 02:54:02,960 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:02,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:02,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:02,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:02,964 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:54:02,964 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:54:02,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:54:02,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:54:02,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:54:02,964 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:54:02,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:54:02,964 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:02,964 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-04-12 02:54:02,964 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:54:02,964 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:54:02,964 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:54:02,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:54:02,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:54:02,964 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:54:02,965 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:54:02,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:54:02,964 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testSplitWithoutPONR' 2023-04-12 02:54:02,965 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:54:02,964 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:02,965 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:02,965 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-04-12 02:54:02,965 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:02,965 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-12 02:54:02,965 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:54:02,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:54:02,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:54:02,965 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 02:54:02,965 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:02,966 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:02,966 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-12 02:54:02,966 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-12 02:54:02,966 DEBUG [Listener at localhost/46667] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testSplitWithoutPONR'' to complete. (max 20000 ms per retry) 2023-04-12 02:54:02,966 DEBUG [Listener at localhost/46667] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 02:54:03,066 DEBUG [Listener at localhost/46667] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 02:54:03,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 02:54:03,099 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 02:54:03,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] procedure.ProcedureCoordinator(143): Procedure testSplitWithoutPONR was in running list but was completed. Accepting new attempt. 2023-04-12 02:54:03,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] procedure.ProcedureCoordinator(165): Submitting procedure testSplitWithoutPONR 2023-04-12 02:54:03,101 INFO [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testSplitWithoutPONR' 2023-04-12 02:54:03,101 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:54:03,102 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testSplitWithoutPONR' starting 'acquire' 2023-04-12 02:54:03,102 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testSplitWithoutPONR', kicking off acquire phase on members. 2023-04-12 02:54:03,103 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-12 02:54:03,103 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-04-12 02:54:03,104 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:03,104 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:54:03,104 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:54:03,104 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:54:03,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:54:03,104 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:54:03,105 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:03,105 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:54:03,105 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 02:54:03,104 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:54:03,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:54:03,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-04-12 02:54:03,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:54:03,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-04-12 02:54:03,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-04-12 02:54:03,105 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-12 02:54:03,105 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-12 02:54:03,105 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-12 02:54:03,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 02:54:03,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-04-12 02:54:03,106 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitWithoutPONR 2023-04-12 02:54:03,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 02:54:03,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 02:54:03,106 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitWithoutPONR is still present, removing 2023-04-12 02:54:03,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-04-12 02:54:03,106 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitWithoutPONR 2023-04-12 02:54:03,106 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitWithoutPONR 2023-04-12 02:54:03,106 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitWithoutPONR' with timeout 60000ms 2023-04-12 02:54:03,106 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:54:03,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-04-12 02:54:03,107 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitWithoutPONR' starting 'acquire' stage 2023-04-12 02:54:03,107 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitWithoutPONR 2023-04-12 02:54:03,107 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitWithoutPONR is still present, removing 2023-04-12 02:54:03,107 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitWithoutPONR 2023-04-12 02:54:03,107 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitWithoutPONR' with timeout 60000ms 2023-04-12 02:54:03,107 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:54:03,108 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitWithoutPONR' starting 'acquire' stage 2023-04-12 02:54:03,108 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 02:54:03,108 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 02:54:03,108 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitWithoutPONR' locally acquired 2023-04-12 02:54:03,108 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44679,1681268031772' joining acquired barrier for procedure (testSplitWithoutPONR) in zk 2023-04-12 02:54:03,108 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitWithoutPONR is still present, removing 2023-04-12 02:54:03,109 DEBUG [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testSplitWithoutPONR,,1681268041279.315e13e868a7f68a3ce37802b2f6312d. 2023-04-12 02:54:03,109 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitWithoutPONR 2023-04-12 02:54:03,109 DEBUG [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testSplitWithoutPONR,,1681268041279.315e13e868a7f68a3ce37802b2f6312d. started... 2023-04-12 02:54:03,109 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitWithoutPONR' with timeout 60000ms 2023-04-12 02:54:03,109 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:54:03,111 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitWithoutPONR' starting 'acquire' stage 2023-04-12 02:54:03,111 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitWithoutPONR' locally acquired 2023-04-12 02:54:03,111 INFO [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 315e13e868a7f68a3ce37802b2f6312d 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-04-12 02:54:03,111 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43091,1681268031804' joining acquired barrier for procedure (testSplitWithoutPONR) in zk 2023-04-12 02:54:03,118 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-12 02:54:03,120 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-12 02:54:03,120 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-12 02:54:03,120 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitWithoutPONR' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 02:54:03,120 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-12 02:54:03,120 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitWithoutPONR' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 02:54:03,147 INFO [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=45 (bloomFilter=true), to=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/.tmp/cf1/7e0f8c9c61e247ecbfc4c27206fae340 2023-04-12 02:54:03,184 INFO [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=45 (bloomFilter=true), to=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/.tmp/cf2/3a80c419c6b94682b44cf75bc8aef7bd 2023-04-12 02:54:03,192 DEBUG [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/.tmp/cf1/7e0f8c9c61e247ecbfc4c27206fae340 as hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf1/7e0f8c9c61e247ecbfc4c27206fae340 2023-04-12 02:54:03,204 INFO [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf1/7e0f8c9c61e247ecbfc4c27206fae340, entries=10, sequenceid=45, filesize=5.0 K 2023-04-12 02:54:03,205 DEBUG [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/.tmp/cf2/3a80c419c6b94682b44cf75bc8aef7bd as hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf2/3a80c419c6b94682b44cf75bc8aef7bd 2023-04-12 02:54:03,213 INFO [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf2/3a80c419c6b94682b44cf75bc8aef7bd, entries=10, sequenceid=45, filesize=5.0 K 2023-04-12 02:54:03,215 INFO [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for 315e13e868a7f68a3ce37802b2f6312d in 106ms, sequenceid=45, compaction requested=true 2023-04-12 02:54:03,216 DEBUG [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 315e13e868a7f68a3ce37802b2f6312d: 2023-04-12 02:54:03,216 DEBUG [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testSplitWithoutPONR,,1681268041279.315e13e868a7f68a3ce37802b2f6312d. 2023-04-12 02:54:03,216 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 02:54:03,216 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 02:54:03,216 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:03,216 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitWithoutPONR' locally acquired 2023-04-12 02:54:03,216 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34579,1681268031719' joining acquired barrier for procedure (testSplitWithoutPONR) in zk 2023-04-12 02:54:03,220 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:03,220 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-12 02:54:03,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:03,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 02:54:03,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 02:54:03,221 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-12 02:54:03,221 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitWithoutPONR' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 02:54:03,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 02:54:03,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 02:54:03,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitWithoutPONR 2023-04-12 02:54:03,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:03,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:03,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:03,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 02:54:03,225 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34579,1681268031719' joining acquired barrier for procedure 'testSplitWithoutPONR' on coordinator 2023-04-12 02:54:03,225 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1f1ab075[Count = 0] remaining members to acquire global barrier 2023-04-12 02:54:03,225 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testSplitWithoutPONR' starting 'in-barrier' execution. 2023-04-12 02:54:03,225 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-12 02:54:03,226 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-12 02:54:03,226 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-12 02:54:03,226 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-12 02:54:03,226 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-12 02:54:03,226 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-12 02:54:03,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-12 02:54:03,226 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-12 02:54:03,227 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitWithoutPONR' received 'reached' from coordinator. 2023-04-12 02:54:03,227 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitWithoutPONR' locally completed 2023-04-12 02:54:03,227 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitWithoutPONR' completed for member 'jenkins-hbase4.apache.org,43091,1681268031804' in zk 2023-04-12 02:54:03,227 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:03,227 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 02:54:03,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-12 02:54:03,227 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitWithoutPONR' received 'reached' from coordinator. 2023-04-12 02:54:03,227 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitWithoutPONR' locally completed 2023-04-12 02:54:03,227 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitWithoutPONR' completed for member 'jenkins-hbase4.apache.org,44679,1681268031772' in zk 2023-04-12 02:54:03,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-12 02:54:03,227 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitWithoutPONR' received 'reached' from coordinator. 2023-04-12 02:54:03,227 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitWithoutPONR' locally completed 2023-04-12 02:54:03,227 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitWithoutPONR' completed for member 'jenkins-hbase4.apache.org,34579,1681268031719' in zk 2023-04-12 02:54:03,230 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitWithoutPONR' has notified controller of completion 2023-04-12 02:54:03,230 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:54:03,230 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitWithoutPONR' completed. 2023-04-12 02:54:03,230 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitWithoutPONR' has notified controller of completion 2023-04-12 02:54:03,230 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:54:03,230 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitWithoutPONR' completed. 2023-04-12 02:54:03,231 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitWithoutPONR' has notified controller of completion 2023-04-12 02:54:03,231 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:54:03,231 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitWithoutPONR' completed. 2023-04-12 02:54:03,231 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:03,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:03,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 02:54:03,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 02:54:03,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 02:54:03,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 02:54:03,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitWithoutPONR 2023-04-12 02:54:03,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:03,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:03,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:03,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 02:54:03,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitWithoutPONR 2023-04-12 02:54:03,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:03,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:03,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:03,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testSplitWithoutPONR' member 'jenkins-hbase4.apache.org,34579,1681268031719': 2023-04-12 02:54:03,238 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34579,1681268031719' released barrier for procedure'testSplitWithoutPONR', counting down latch. Waiting for 0 more 2023-04-12 02:54:03,238 INFO [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testSplitWithoutPONR' execution completed 2023-04-12 02:54:03,238 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 02:54:03,238 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 02:54:03,238 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testSplitWithoutPONR 2023-04-12 02:54:03,238 INFO [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testSplitWithoutPONRincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 02:54:03,240 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-12 02:54:03,240 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-12 02:54:03,240 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-12 02:54:03,240 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-12 02:54:03,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-12 02:54:03,240 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-12 02:54:03,240 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:54:03,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-12 02:54:03,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 02:54:03,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 02:54:03,240 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:54:03,240 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-12 02:54:03,241 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:54:03,241 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-12 02:54:03,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-12 02:54:03,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-12 02:54:03,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:54:03,241 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:54:03,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 02:54:03,241 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:03,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-12 02:54:03,242 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:54:03,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:54:03,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitWithoutPONR 2023-04-12 02:54:03,242 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:03,242 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:54:03,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:54:03,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 02:54:03,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-12 02:54:03,243 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:03,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-12 02:54:03,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitWithoutPONR 2023-04-12 02:54:03,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:03,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:03,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:03,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 02:54:03,245 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:03,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitWithoutPONR 2023-04-12 02:54:03,245 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:03,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:03,246 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:03,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:03,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:03,251 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:54:03,251 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:54:03,251 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:03,251 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:54:03,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:54:03,251 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-04-12 02:54:03,251 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:03,251 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:03,251 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-04-12 02:54:03,251 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:03,251 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-12 02:54:03,251 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:03,251 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:03,251 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:54:03,251 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:54:03,251 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:54:03,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:54:03,251 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:54:03,251 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:54:03,252 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:54:03,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:54:03,252 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:54:03,251 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:54:03,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:54:03,251 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testSplitWithoutPONR' 2023-04-12 02:54:03,252 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 02:54:03,252 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:54:03,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:54:03,251 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-12 02:54:03,252 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-12 02:54:03,252 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:54:03,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:54:03,257 DEBUG [Listener at localhost/46667] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testSplitWithoutPONR'' to complete. (max 20000 ms per retry) 2023-04-12 02:54:03,258 DEBUG [Listener at localhost/46667] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 02:54:03,358 DEBUG [Listener at localhost/46667] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 02:54:03,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 02:54:03,382 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 02:54:03,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] procedure.ProcedureCoordinator(143): Procedure testSplitWithoutPONR was in running list but was completed. Accepting new attempt. 2023-04-12 02:54:03,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] procedure.ProcedureCoordinator(165): Submitting procedure testSplitWithoutPONR 2023-04-12 02:54:03,384 INFO [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testSplitWithoutPONR' 2023-04-12 02:54:03,384 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:54:03,384 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testSplitWithoutPONR' starting 'acquire' 2023-04-12 02:54:03,384 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testSplitWithoutPONR', kicking off acquire phase on members. 2023-04-12 02:54:03,385 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-12 02:54:03,385 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-04-12 02:54:03,387 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:03,387 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:54:03,387 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:54:03,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:54:03,387 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:54:03,387 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:54:03,387 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:54:03,387 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:54:03,387 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:03,387 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 02:54:03,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:54:03,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-04-12 02:54:03,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:54:03,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-04-12 02:54:03,388 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-12 02:54:03,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-04-12 02:54:03,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 02:54:03,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-04-12 02:54:03,389 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitWithoutPONR 2023-04-12 02:54:03,389 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitWithoutPONR is still present, removing 2023-04-12 02:54:03,389 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitWithoutPONR 2023-04-12 02:54:03,389 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitWithoutPONR' with timeout 60000ms 2023-04-12 02:54:03,389 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:54:03,390 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitWithoutPONR' starting 'acquire' stage 2023-04-12 02:54:03,390 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-12 02:54:03,390 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 02:54:03,390 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 02:54:03,390 DEBUG [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testSplitWithoutPONR,,1681268041279.315e13e868a7f68a3ce37802b2f6312d. 2023-04-12 02:54:03,390 DEBUG [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testSplitWithoutPONR,,1681268041279.315e13e868a7f68a3ce37802b2f6312d. started... 2023-04-12 02:54:03,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 02:54:03,390 INFO [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 315e13e868a7f68a3ce37802b2f6312d 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-04-12 02:54:03,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-04-12 02:54:03,390 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-12 02:54:03,390 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitWithoutPONR 2023-04-12 02:54:03,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 02:54:03,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-04-12 02:54:03,391 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitWithoutPONR 2023-04-12 02:54:03,396 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitWithoutPONR is still present, removing 2023-04-12 02:54:03,396 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitWithoutPONR 2023-04-12 02:54:03,396 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitWithoutPONR' with timeout 60000ms 2023-04-12 02:54:03,396 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:54:03,396 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitWithoutPONR is still present, removing 2023-04-12 02:54:03,396 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitWithoutPONR' starting 'acquire' stage 2023-04-12 02:54:03,396 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitWithoutPONR 2023-04-12 02:54:03,396 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitWithoutPONR' locally acquired 2023-04-12 02:54:03,396 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitWithoutPONR' with timeout 60000ms 2023-04-12 02:54:03,396 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:54:03,396 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43091,1681268031804' joining acquired barrier for procedure (testSplitWithoutPONR) in zk 2023-04-12 02:54:03,396 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitWithoutPONR' starting 'acquire' stage 2023-04-12 02:54:03,397 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitWithoutPONR' locally acquired 2023-04-12 02:54:03,397 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44679,1681268031772' joining acquired barrier for procedure (testSplitWithoutPONR) in zk 2023-04-12 02:54:03,398 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-12 02:54:03,400 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-12 02:54:03,400 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-12 02:54:03,400 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitWithoutPONR' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 02:54:03,402 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-12 02:54:03,402 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitWithoutPONR' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 02:54:03,421 INFO [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=58 (bloomFilter=true), to=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/.tmp/cf1/2ca08ea408834a75bc73ef0d3d78fa73 2023-04-12 02:54:03,451 INFO [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=58 (bloomFilter=true), to=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/.tmp/cf2/f13ea6a8012d44709becae664369d2d0 2023-04-12 02:54:03,457 DEBUG [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/.tmp/cf1/2ca08ea408834a75bc73ef0d3d78fa73 as hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf1/2ca08ea408834a75bc73ef0d3d78fa73 2023-04-12 02:54:03,466 INFO [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf1/2ca08ea408834a75bc73ef0d3d78fa73, entries=10, sequenceid=58, filesize=5.0 K 2023-04-12 02:54:03,467 DEBUG [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/.tmp/cf2/f13ea6a8012d44709becae664369d2d0 as hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf2/f13ea6a8012d44709becae664369d2d0 2023-04-12 02:54:03,480 INFO [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf2/f13ea6a8012d44709becae664369d2d0, entries=10, sequenceid=58, filesize=5.0 K 2023-04-12 02:54:03,482 INFO [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for 315e13e868a7f68a3ce37802b2f6312d in 92ms, sequenceid=58, compaction requested=true 2023-04-12 02:54:03,482 DEBUG [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 315e13e868a7f68a3ce37802b2f6312d: 2023-04-12 02:54:03,482 DEBUG [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testSplitWithoutPONR,,1681268041279.315e13e868a7f68a3ce37802b2f6312d. 2023-04-12 02:54:03,482 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 02:54:03,482 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 02:54:03,482 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:03,482 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitWithoutPONR' locally acquired 2023-04-12 02:54:03,482 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34579,1681268031719' joining acquired barrier for procedure (testSplitWithoutPONR) in zk 2023-04-12 02:54:03,488 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:03,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:03,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 02:54:03,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 02:54:03,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 02:54:03,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 02:54:03,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitWithoutPONR 2023-04-12 02:54:03,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:03,491 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-12 02:54:03,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:03,491 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-12 02:54:03,491 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitWithoutPONR' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 02:54:03,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:03,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 02:54:03,493 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34579,1681268031719' joining acquired barrier for procedure 'testSplitWithoutPONR' on coordinator 2023-04-12 02:54:03,493 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3d87760b[Count = 0] remaining members to acquire global barrier 2023-04-12 02:54:03,493 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testSplitWithoutPONR' starting 'in-barrier' execution. 2023-04-12 02:54:03,493 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-12 02:54:03,495 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-12 02:54:03,495 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-12 02:54:03,495 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-12 02:54:03,495 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-12 02:54:03,495 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-12 02:54:03,495 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:03,495 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 02:54:03,495 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-12 02:54:03,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-12 02:54:03,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-12 02:54:03,495 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitWithoutPONR' received 'reached' from coordinator. 2023-04-12 02:54:03,495 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitWithoutPONR' received 'reached' from coordinator. 2023-04-12 02:54:03,496 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitWithoutPONR' locally completed 2023-04-12 02:54:03,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-12 02:54:03,496 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitWithoutPONR' completed for member 'jenkins-hbase4.apache.org,34579,1681268031719' in zk 2023-04-12 02:54:03,496 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitWithoutPONR' received 'reached' from coordinator. 2023-04-12 02:54:03,496 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitWithoutPONR' locally completed 2023-04-12 02:54:03,496 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitWithoutPONR' completed for member 'jenkins-hbase4.apache.org,43091,1681268031804' in zk 2023-04-12 02:54:03,495 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitWithoutPONR' locally completed 2023-04-12 02:54:03,496 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitWithoutPONR' completed for member 'jenkins-hbase4.apache.org,44679,1681268031772' in zk 2023-04-12 02:54:03,497 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitWithoutPONR' has notified controller of completion 2023-04-12 02:54:03,497 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:54:03,497 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitWithoutPONR' completed. 2023-04-12 02:54:03,498 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:03,498 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitWithoutPONR' has notified controller of completion 2023-04-12 02:54:03,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:03,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 02:54:03,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 02:54:03,498 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:54:03,498 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitWithoutPONR' completed. 2023-04-12 02:54:03,498 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitWithoutPONR' has notified controller of completion 2023-04-12 02:54:03,499 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:54:03,499 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitWithoutPONR' completed. 2023-04-12 02:54:03,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 02:54:03,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 02:54:03,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitWithoutPONR 2023-04-12 02:54:03,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:03,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:03,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:03,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 02:54:03,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitWithoutPONR 2023-04-12 02:54:03,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:03,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:03,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:03,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testSplitWithoutPONR' member 'jenkins-hbase4.apache.org,34579,1681268031719': 2023-04-12 02:54:03,503 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34579,1681268031719' released barrier for procedure'testSplitWithoutPONR', counting down latch. Waiting for 0 more 2023-04-12 02:54:03,503 INFO [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testSplitWithoutPONR' execution completed 2023-04-12 02:54:03,503 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 02:54:03,503 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 02:54:03,503 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testSplitWithoutPONR 2023-04-12 02:54:03,503 INFO [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testSplitWithoutPONRincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 02:54:03,508 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-12 02:54:03,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-12 02:54:03,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 02:54:03,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 02:54:03,509 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-12 02:54:03,509 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:54:03,509 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-12 02:54:03,509 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-12 02:54:03,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-12 02:54:03,509 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-12 02:54:03,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-12 02:54:03,509 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-12 02:54:03,509 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:54:03,509 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-12 02:54:03,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-12 02:54:03,509 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:54:03,510 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:54:03,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:54:03,510 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:54:03,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:54:03,510 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:54:03,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:54:03,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 02:54:03,510 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:03,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-12 02:54:03,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-12 02:54:03,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-12 02:54:03,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitWithoutPONR 2023-04-12 02:54:03,512 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:03,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 02:54:03,513 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:03,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitWithoutPONR 2023-04-12 02:54:03,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:03,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:03,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:03,516 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:03,517 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:03,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 02:54:03,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitWithoutPONR 2023-04-12 02:54:03,519 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:03,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:03,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:03,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:03,525 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:54:03,525 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:54:03,525 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:54:03,525 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:54:03,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:54:03,525 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:03,525 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:54:03,525 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:54:03,525 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:54:03,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:54:03,525 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:54:03,525 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-04-12 02:54:03,525 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:54:03,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:54:03,526 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:54:03,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:54:03,526 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:54:03,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:54:03,526 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:03,526 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:03,526 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-04-12 02:54:03,526 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:03,526 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-12 02:54:03,526 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:03,526 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:03,526 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:54:03,526 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-12 02:54:03,526 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-12 02:54:03,526 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:54:03,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:54:03,526 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testSplitWithoutPONR' 2023-04-12 02:54:03,527 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 02:54:03,527 DEBUG [Listener at localhost/46667] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testSplitWithoutPONR'' to complete. (max 20000 ms per retry) 2023-04-12 02:54:03,527 DEBUG [Listener at localhost/46667] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 02:54:03,627 DEBUG [Listener at localhost/46667] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 02:54:03,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 02:54:03,667 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 02:54:03,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] procedure.ProcedureCoordinator(143): Procedure testSplitWithoutPONR was in running list but was completed. Accepting new attempt. 2023-04-12 02:54:03,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] procedure.ProcedureCoordinator(165): Submitting procedure testSplitWithoutPONR 2023-04-12 02:54:03,670 INFO [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testSplitWithoutPONR' 2023-04-12 02:54:03,670 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:54:03,670 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testSplitWithoutPONR' starting 'acquire' 2023-04-12 02:54:03,670 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testSplitWithoutPONR', kicking off acquire phase on members. 2023-04-12 02:54:03,671 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-12 02:54:03,671 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-04-12 02:54:03,672 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:54:03,672 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:54:03,673 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:54:03,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:54:03,673 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:54:03,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:54:03,674 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:03,676 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:54:03,676 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:54:03,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-04-12 02:54:03,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:54:03,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-04-12 02:54:03,676 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:03,676 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 02:54:03,676 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-12 02:54:03,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-04-12 02:54:03,676 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-12 02:54:03,677 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-12 02:54:03,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 02:54:03,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 02:54:03,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-04-12 02:54:03,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-04-12 02:54:03,678 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitWithoutPONR 2023-04-12 02:54:03,678 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitWithoutPONR 2023-04-12 02:54:03,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 02:54:03,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-04-12 02:54:03,678 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitWithoutPONR 2023-04-12 02:54:03,680 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitWithoutPONR is still present, removing 2023-04-12 02:54:03,680 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitWithoutPONR is still present, removing 2023-04-12 02:54:03,680 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitWithoutPONR 2023-04-12 02:54:03,680 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitWithoutPONR 2023-04-12 02:54:03,680 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitWithoutPONR' with timeout 60000ms 2023-04-12 02:54:03,680 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitWithoutPONR' with timeout 60000ms 2023-04-12 02:54:03,680 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:54:03,680 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:54:03,680 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitWithoutPONR' starting 'acquire' stage 2023-04-12 02:54:03,680 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitWithoutPONR' starting 'acquire' stage 2023-04-12 02:54:03,680 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitWithoutPONR' locally acquired 2023-04-12 02:54:03,680 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44679,1681268031772' joining acquired barrier for procedure (testSplitWithoutPONR) in zk 2023-04-12 02:54:03,683 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-12 02:54:03,683 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-12 02:54:03,683 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitWithoutPONR' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 02:54:03,684 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitWithoutPONR is still present, removing 2023-04-12 02:54:03,684 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitWithoutPONR 2023-04-12 02:54:03,684 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 02:54:03,684 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitWithoutPONR' with timeout 60000ms 2023-04-12 02:54:03,684 DEBUG [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testSplitWithoutPONR,,1681268041279.315e13e868a7f68a3ce37802b2f6312d. 2023-04-12 02:54:03,684 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:54:03,684 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 02:54:03,684 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitWithoutPONR' starting 'acquire' stage 2023-04-12 02:54:03,684 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitWithoutPONR' locally acquired 2023-04-12 02:54:03,684 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43091,1681268031804' joining acquired barrier for procedure (testSplitWithoutPONR) in zk 2023-04-12 02:54:03,684 DEBUG [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testSplitWithoutPONR,,1681268041279.315e13e868a7f68a3ce37802b2f6312d. started... 2023-04-12 02:54:03,685 INFO [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 315e13e868a7f68a3ce37802b2f6312d 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-04-12 02:54:03,686 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-12 02:54:03,686 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-12 02:54:03,686 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitWithoutPONR' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 02:54:03,731 INFO [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=71 (bloomFilter=true), to=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/.tmp/cf1/713becfd985e40a8892fbc1f6fe6bdb4 2023-04-12 02:54:04,165 INFO [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=71 (bloomFilter=true), to=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/.tmp/cf2/d5ab6ea45d1b4298ae7d38df0de5a43f 2023-04-12 02:54:04,175 DEBUG [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/.tmp/cf1/713becfd985e40a8892fbc1f6fe6bdb4 as hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf1/713becfd985e40a8892fbc1f6fe6bdb4 2023-04-12 02:54:04,190 INFO [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf1/713becfd985e40a8892fbc1f6fe6bdb4, entries=10, sequenceid=71, filesize=5.0 K 2023-04-12 02:54:04,191 DEBUG [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/.tmp/cf2/d5ab6ea45d1b4298ae7d38df0de5a43f as hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf2/d5ab6ea45d1b4298ae7d38df0de5a43f 2023-04-12 02:54:04,214 INFO [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf2/d5ab6ea45d1b4298ae7d38df0de5a43f, entries=10, sequenceid=71, filesize=5.0 K 2023-04-12 02:54:04,220 INFO [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for 315e13e868a7f68a3ce37802b2f6312d in 535ms, sequenceid=71, compaction requested=true 2023-04-12 02:54:04,220 DEBUG [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 315e13e868a7f68a3ce37802b2f6312d: 2023-04-12 02:54:04,220 DEBUG [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testSplitWithoutPONR,,1681268041279.315e13e868a7f68a3ce37802b2f6312d. 2023-04-12 02:54:04,220 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 02:54:04,220 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 02:54:04,220 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:04,220 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitWithoutPONR' locally acquired 2023-04-12 02:54:04,220 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34579,1681268031719' joining acquired barrier for procedure (testSplitWithoutPONR) in zk 2023-04-12 02:54:04,222 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-12 02:54:04,222 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-12 02:54:04,222 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitWithoutPONR' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 02:54:04,223 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:04,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:04,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 02:54:04,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 02:54:04,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 02:54:04,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 02:54:04,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitWithoutPONR 2023-04-12 02:54:04,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:04,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:04,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:04,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 02:54:04,228 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34579,1681268031719' joining acquired barrier for procedure 'testSplitWithoutPONR' on coordinator 2023-04-12 02:54:04,228 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@fef3449[Count = 0] remaining members to acquire global barrier 2023-04-12 02:54:04,228 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testSplitWithoutPONR' starting 'in-barrier' execution. 2023-04-12 02:54:04,228 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-12 02:54:04,230 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-12 02:54:04,230 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-12 02:54:04,230 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-12 02:54:04,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-12 02:54:04,230 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitWithoutPONR' received 'reached' from coordinator. 2023-04-12 02:54:04,230 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-12 02:54:04,230 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-12 02:54:04,230 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:04,231 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 02:54:04,230 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitWithoutPONR' locally completed 2023-04-12 02:54:04,230 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-12 02:54:04,231 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitWithoutPONR' completed for member 'jenkins-hbase4.apache.org,44679,1681268031772' in zk 2023-04-12 02:54:04,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-12 02:54:04,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-12 02:54:04,231 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitWithoutPONR' received 'reached' from coordinator. 2023-04-12 02:54:04,231 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitWithoutPONR' received 'reached' from coordinator. 2023-04-12 02:54:04,231 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitWithoutPONR' locally completed 2023-04-12 02:54:04,231 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitWithoutPONR' locally completed 2023-04-12 02:54:04,231 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitWithoutPONR' completed for member 'jenkins-hbase4.apache.org,43091,1681268031804' in zk 2023-04-12 02:54:04,231 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitWithoutPONR' completed for member 'jenkins-hbase4.apache.org,34579,1681268031719' in zk 2023-04-12 02:54:04,236 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitWithoutPONR' has notified controller of completion 2023-04-12 02:54:04,236 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:54:04,236 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitWithoutPONR' completed. 2023-04-12 02:54:04,237 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitWithoutPONR' has notified controller of completion 2023-04-12 02:54:04,237 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:54:04,237 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitWithoutPONR' completed. 2023-04-12 02:54:04,238 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitWithoutPONR' has notified controller of completion 2023-04-12 02:54:04,238 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:54:04,238 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:04,238 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitWithoutPONR' completed. 2023-04-12 02:54:04,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:04,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 02:54:04,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 02:54:04,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 02:54:04,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 02:54:04,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitWithoutPONR 2023-04-12 02:54:04,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:04,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:04,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:04,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 02:54:04,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitWithoutPONR 2023-04-12 02:54:04,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:04,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:04,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:04,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testSplitWithoutPONR' member 'jenkins-hbase4.apache.org,34579,1681268031719': 2023-04-12 02:54:04,249 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34579,1681268031719' released barrier for procedure'testSplitWithoutPONR', counting down latch. Waiting for 0 more 2023-04-12 02:54:04,249 INFO [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testSplitWithoutPONR' execution completed 2023-04-12 02:54:04,249 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 02:54:04,249 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 02:54:04,249 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testSplitWithoutPONR 2023-04-12 02:54:04,249 INFO [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testSplitWithoutPONRincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 02:54:04,251 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-12 02:54:04,251 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:54:04,251 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-12 02:54:04,251 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-12 02:54:04,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-12 02:54:04,252 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-12 02:54:04,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-12 02:54:04,252 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:54:04,252 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-12 02:54:04,252 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-12 02:54:04,252 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:54:04,252 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:54:04,252 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:54:04,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:54:04,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-12 02:54:04,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 02:54:04,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 02:54:04,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:54:04,252 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-12 02:54:04,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-12 02:54:04,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-12 02:54:04,253 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:54:04,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:54:04,254 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:04,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-12 02:54:04,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 02:54:04,254 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:04,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-12 02:54:04,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitWithoutPONR 2023-04-12 02:54:04,255 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:04,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 02:54:04,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitWithoutPONR 2023-04-12 02:54:04,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:04,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:04,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:04,257 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:04,258 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:04,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 02:54:04,258 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:04,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitWithoutPONR 2023-04-12 02:54:04,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:04,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:04,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:04,264 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:54:04,264 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:04,264 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:54:04,264 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-04-12 02:54:04,264 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:04,264 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:04,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:54:04,265 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:54:04,265 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:54:04,265 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-04-12 02:54:04,265 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:54:04,265 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:54:04,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:54:04,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:54:04,265 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:54:04,265 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testSplitWithoutPONR' 2023-04-12 02:54:04,265 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 02:54:04,265 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:54:04,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:54:04,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:54:04,265 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:04,266 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-12 02:54:04,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:54:04,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:54:04,266 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:04,266 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:54:04,266 DEBUG [Listener at localhost/46667] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testSplitWithoutPONR'' to complete. (max 20000 ms per retry) 2023-04-12 02:54:04,266 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:04,266 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-12 02:54:04,266 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-12 02:54:04,266 DEBUG [Listener at localhost/46667] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 02:54:04,266 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:54:04,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:54:04,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:54:04,367 DEBUG [Listener at localhost/46667] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 02:54:04,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 02:54:04,632 WARN [Listener at localhost/46667] procedure2.ProcedureTestingUtility(193): Set Kill before store update to: true 2023-04-12 02:54:04,634 DEBUG [Listener at localhost/46667] assignment.SplitTableRegionProcedure(219): Splittable=true state=OPEN, location=jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:04,635 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(1029): Stored pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=315e13e868a7f68a3ce37802b2f6312d, daughterA=4e1092f3c00484f8f09c730861225d43, daughterB=291734c00d212fd203b04cf8872d4649 2023-04-12 02:54:04,636 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=315e13e868a7f68a3ce37802b2f6312d, daughterA=4e1092f3c00484f8f09c730861225d43, daughterB=291734c00d212fd203b04cf8872d4649 2023-04-12 02:54:04,636 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=315e13e868a7f68a3ce37802b2f6312d, daughterA=4e1092f3c00484f8f09c730861225d43, daughterB=291734c00d212fd203b04cf8872d4649 2023-04-12 02:54:04,636 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=315e13e868a7f68a3ce37802b2f6312d, daughterA=4e1092f3c00484f8f09c730861225d43, daughterB=291734c00d212fd203b04cf8872d4649 2023-04-12 02:54:04,637 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-04-12 02:54:04,638 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION, locked=true; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=315e13e868a7f68a3ce37802b2f6312d, daughterA=4e1092f3c00484f8f09c730861225d43, daughterB=291734c00d212fd203b04cf8872d4649 2023-04-12 02:54:04,638 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-04-12 02:54:04,638 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION, locked=true; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=315e13e868a7f68a3ce37802b2f6312d, daughterA=4e1092f3c00484f8f09c730861225d43, daughterB=291734c00d212fd203b04cf8872d4649 at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-04-12 02:54:04,681 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-12 02:54:04,886 INFO [Listener at localhost/46667] procedure.MasterProcedureTestingUtility(417): Restart 2 exec state=pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION, locked=true; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=315e13e868a7f68a3ce37802b2f6312d, daughterA=4e1092f3c00484f8f09c730861225d43, daughterB=291734c00d212fd203b04cf8872d4649 2023-04-12 02:54:04,887 INFO [Listener at localhost/46667] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-04-12 02:54:04,887 INFO [Listener at localhost/46667] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-04-12 02:54:04,888 INFO [Listener at localhost/46667] assignment.AssignmentManager(315): Stopping assignment manager 2023-04-12 02:54:04,890 ERROR [Listener at localhost/46667] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] Thread[HFileArchiver-1,5,PEWorkerGroup] Thread[HFileArchiver-2,5,PEWorkerGroup] Thread[HFileArchiver-3,5,PEWorkerGroup] Thread[HFileArchiver-4,5,PEWorkerGroup] Thread[HFileArchiver-5,5,PEWorkerGroup] Thread[HFileArchiver-6,5,PEWorkerGroup] 2023-04-12 02:54:04,891 INFO [Listener at localhost/46667] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-04-12 02:54:04,891 INFO [Listener at localhost/46667] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-04-12 02:54:04,891 INFO [Listener at localhost/46667] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-04-12 02:54:04,892 INFO [Listener at localhost/46667] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-04-12 02:54:04,893 INFO [Listener at localhost/46667] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-04-12 02:54:04,896 DEBUG [Listener at localhost/46667] store.ProcedureTree(114): Procedure Procedure(pid=28, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[] 2023-04-12 02:54:04,898 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-04-12 02:54:04,899 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-04-12 02:54:04,899 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-04-12 02:54:04,899 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-04-12 02:54:04,900 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-12 02:54:04,900 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=cc9012ec0b656baa24d04368fbe026c4, daughterA=e2a9b8fe15a781798c0b49e48ff5aad7, daughterB=6f55b5d6a460bd8c7569dee22f049104 2023-04-12 02:54:04,901 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-12 02:54:04,901 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-12 02:54:04,902 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-04-12 02:54:04,902 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(468): Loading pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=315e13e868a7f68a3ce37802b2f6312d, daughterA=4e1092f3c00484f8f09c730861225d43, daughterB=291734c00d212fd203b04cf8872d4649 2023-04-12 02:54:04,903 DEBUG [Listener at localhost/46667] procedure2.Procedure(966): pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=315e13e868a7f68a3ce37802b2f6312d, daughterA=4e1092f3c00484f8f09c730861225d43, daughterB=291734c00d212fd203b04cf8872d4649 held the lock before restarting, call acquireLock to restore it. 2023-04-12 02:54:04,903 INFO [Listener at localhost/46667] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=315e13e868a7f68a3ce37802b2f6312d, daughterA=4e1092f3c00484f8f09c730861225d43, daughterB=291734c00d212fd203b04cf8872d4649 2023-04-12 02:54:04,903 INFO [Listener at localhost/46667] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=315e13e868a7f68a3ce37802b2f6312d, daughterA=4e1092f3c00484f8f09c730861225d43, daughterB=291734c00d212fd203b04cf8872d4649 2023-04-12 02:54:04,903 INFO [Listener at localhost/46667] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=315e13e868a7f68a3ce37802b2f6312d, daughterA=4e1092f3c00484f8f09c730861225d43, daughterB=291734c00d212fd203b04cf8872d4649 2023-04-12 02:54:04,904 INFO [Listener at localhost/46667] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 10 msec 2023-04-12 02:54:04,905 DEBUG [Listener at localhost/46667] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-04-12 02:54:04,906 DEBUG [Listener at localhost/46667] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase4.apache.org,44679,1681268031772, table=hbase:meta, region=1588230740 2023-04-12 02:54:04,916 INFO [Listener at localhost/46667] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1681268074916 2023-04-12 02:54:04,917 DEBUG [Listener at localhost/46667] assignment.AssignmentManager(1517): Joining cluster... 2023-04-12 02:54:04,919 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=315e13e868a7f68a3ce37802b2f6312d, daughterA=4e1092f3c00484f8f09c730861225d43, daughterB=291734c00d212fd203b04cf8872d4649 2023-04-12 02:54:04,929 DEBUG [Listener at localhost/46667] assignment.RegionStateStore(147): Load hbase:meta entry region=1f255e980a31bb266f2206a30aa2ae3f, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,44679,1681268031772, regionLocation=jenkins-hbase4.apache.org,44679,1681268031772, openSeqNum=2 2023-04-12 02:54:04,930 DEBUG [Listener at localhost/46667] assignment.RegionStateStore(147): Load hbase:meta entry region=315e13e868a7f68a3ce37802b2f6312d, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,34579,1681268031719, regionLocation=jenkins-hbase4.apache.org,34579,1681268031719, openSeqNum=2 2023-04-12 02:54:04,930 INFO [Listener at localhost/46667] assignment.AssignmentManager(1529): Number of RegionServers=3 2023-04-12 02:54:04,930 INFO [Listener at localhost/46667] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1681268104930 2023-04-12 02:54:04,930 INFO [Listener at localhost/46667] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1681268164930 2023-04-12 02:54:04,930 INFO [Listener at localhost/46667] assignment.AssignmentManager(1536): Joined the cluster in 13 msec 2023-04-12 02:54:04,931 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-04-12 02:54:04,934 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-04-12 02:54:04,934 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_CLOSE_PARENT_REGION, locked=true; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=315e13e868a7f68a3ce37802b2f6312d, daughterA=4e1092f3c00484f8f09c730861225d43, daughterB=291734c00d212fd203b04cf8872d4649 2023-04-12 02:54:04,934 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-04-12 02:54:04,934 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_CLOSE_PARENT_REGION, locked=true; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=315e13e868a7f68a3ce37802b2f6312d, daughterA=4e1092f3c00484f8f09c730861225d43, daughterB=291734c00d212fd203b04cf8872d4649 at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-04-12 02:54:05,181 INFO [Listener at localhost/46667] procedure.MasterProcedureTestingUtility(417): Restart 3 exec state=pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_CLOSE_PARENT_REGION, locked=true; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=315e13e868a7f68a3ce37802b2f6312d, daughterA=4e1092f3c00484f8f09c730861225d43, daughterB=291734c00d212fd203b04cf8872d4649 2023-04-12 02:54:05,181 INFO [Listener at localhost/46667] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-04-12 02:54:05,181 INFO [Listener at localhost/46667] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-04-12 02:54:05,181 INFO [Listener at localhost/46667] assignment.AssignmentManager(315): Stopping assignment manager 2023-04-12 02:54:05,182 ERROR [Listener at localhost/46667] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-04-12 02:54:05,182 INFO [Listener at localhost/46667] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-04-12 02:54:05,182 INFO [Listener at localhost/46667] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-04-12 02:54:05,182 INFO [Listener at localhost/46667] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-04-12 02:54:05,182 INFO [Listener at localhost/46667] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-04-12 02:54:05,184 INFO [Listener at localhost/46667] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 1 msec 2023-04-12 02:54:05,186 DEBUG [Listener at localhost/46667] store.ProcedureTree(114): Procedure Procedure(pid=28, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[0] 2023-04-12 02:54:05,186 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-04-12 02:54:05,187 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-04-12 02:54:05,187 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-04-12 02:54:05,188 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-04-12 02:54:05,188 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-12 02:54:05,189 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=cc9012ec0b656baa24d04368fbe026c4, daughterA=e2a9b8fe15a781798c0b49e48ff5aad7, daughterB=6f55b5d6a460bd8c7569dee22f049104 2023-04-12 02:54:05,189 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-12 02:54:05,189 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-12 02:54:05,190 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-04-12 02:54:05,190 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(468): Loading pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=315e13e868a7f68a3ce37802b2f6312d, daughterA=4e1092f3c00484f8f09c730861225d43, daughterB=291734c00d212fd203b04cf8872d4649 2023-04-12 02:54:05,190 DEBUG [Listener at localhost/46667] procedure2.Procedure(966): pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=315e13e868a7f68a3ce37802b2f6312d, daughterA=4e1092f3c00484f8f09c730861225d43, daughterB=291734c00d212fd203b04cf8872d4649 held the lock before restarting, call acquireLock to restore it. 2023-04-12 02:54:05,191 INFO [Listener at localhost/46667] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=315e13e868a7f68a3ce37802b2f6312d, daughterA=4e1092f3c00484f8f09c730861225d43, daughterB=291734c00d212fd203b04cf8872d4649 2023-04-12 02:54:05,191 INFO [Listener at localhost/46667] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=315e13e868a7f68a3ce37802b2f6312d, daughterA=4e1092f3c00484f8f09c730861225d43, daughterB=291734c00d212fd203b04cf8872d4649 2023-04-12 02:54:05,191 INFO [Listener at localhost/46667] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=315e13e868a7f68a3ce37802b2f6312d, daughterA=4e1092f3c00484f8f09c730861225d43, daughterB=291734c00d212fd203b04cf8872d4649 2023-04-12 02:54:05,191 INFO [Listener at localhost/46667] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 7 msec 2023-04-12 02:54:05,192 DEBUG [Listener at localhost/46667] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-04-12 02:54:05,193 DEBUG [Listener at localhost/46667] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase4.apache.org,44679,1681268031772, table=hbase:meta, region=1588230740 2023-04-12 02:54:05,193 INFO [Listener at localhost/46667] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1681268075193 2023-04-12 02:54:05,194 DEBUG [Listener at localhost/46667] assignment.AssignmentManager(1517): Joining cluster... 2023-04-12 02:54:05,196 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=315e13e868a7f68a3ce37802b2f6312d, daughterA=4e1092f3c00484f8f09c730861225d43, daughterB=291734c00d212fd203b04cf8872d4649 2023-04-12 02:54:05,200 DEBUG [Listener at localhost/46667] assignment.RegionStateStore(147): Load hbase:meta entry region=1f255e980a31bb266f2206a30aa2ae3f, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,44679,1681268031772, regionLocation=jenkins-hbase4.apache.org,44679,1681268031772, openSeqNum=2 2023-04-12 02:54:05,200 DEBUG [Listener at localhost/46667] assignment.RegionStateStore(147): Load hbase:meta entry region=315e13e868a7f68a3ce37802b2f6312d, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,34579,1681268031719, regionLocation=jenkins-hbase4.apache.org,34579,1681268031719, openSeqNum=2 2023-04-12 02:54:05,200 INFO [Listener at localhost/46667] assignment.AssignmentManager(1529): Number of RegionServers=3 2023-04-12 02:54:05,201 INFO [Listener at localhost/46667] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1681268105201 2023-04-12 02:54:05,201 INFO [Listener at localhost/46667] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1681268165201 2023-04-12 02:54:05,201 INFO [Listener at localhost/46667] assignment.AssignmentManager(1536): Joined the cluster in 7 msec 2023-04-12 02:54:05,201 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-04-12 02:54:05,206 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitWithoutPONR, region=315e13e868a7f68a3ce37802b2f6312d, UNASSIGN}] 2023-04-12 02:54:05,206 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-04-12 02:54:05,206 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=28, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS, locked=true; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=315e13e868a7f68a3ce37802b2f6312d, daughterA=4e1092f3c00484f8f09c730861225d43, daughterB=291734c00d212fd203b04cf8872d4649 2023-04-12 02:54:05,206 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-04-12 02:54:05,206 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=28, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS, locked=true; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=315e13e868a7f68a3ce37802b2f6312d, daughterA=4e1092f3c00484f8f09c730861225d43, daughterB=291734c00d212fd203b04cf8872d4649 at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-04-12 02:54:05,451 INFO [Listener at localhost/46667] procedure.MasterProcedureTestingUtility(417): Restart 11 exec state=pid=28, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS, locked=true; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=315e13e868a7f68a3ce37802b2f6312d, daughterA=4e1092f3c00484f8f09c730861225d43, daughterB=291734c00d212fd203b04cf8872d4649 2023-04-12 02:54:05,451 INFO [Listener at localhost/46667] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-04-12 02:54:05,451 INFO [Listener at localhost/46667] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-04-12 02:54:05,451 INFO [Listener at localhost/46667] assignment.AssignmentManager(315): Stopping assignment manager 2023-04-12 02:54:05,453 ERROR [Listener at localhost/46667] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-04-12 02:54:05,453 INFO [Listener at localhost/46667] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-04-12 02:54:05,453 INFO [Listener at localhost/46667] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-04-12 02:54:05,453 INFO [Listener at localhost/46667] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-04-12 02:54:05,453 INFO [Listener at localhost/46667] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-04-12 02:54:05,454 INFO [Listener at localhost/46667] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-04-12 02:54:05,456 DEBUG [Listener at localhost/46667] store.ProcedureTree(114): Procedure Procedure(pid=28, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[0, 1] 2023-04-12 02:54:05,456 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-04-12 02:54:05,457 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-04-12 02:54:05,457 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-04-12 02:54:05,457 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-04-12 02:54:05,458 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-12 02:54:05,459 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=cc9012ec0b656baa24d04368fbe026c4, daughterA=e2a9b8fe15a781798c0b49e48ff5aad7, daughterB=6f55b5d6a460bd8c7569dee22f049104 2023-04-12 02:54:05,459 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-12 02:54:05,460 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-12 02:54:05,460 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-04-12 02:54:05,461 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(468): Loading pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_CLOSE_PARENT_REGION; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=315e13e868a7f68a3ce37802b2f6312d, daughterA=4e1092f3c00484f8f09c730861225d43, daughterB=291734c00d212fd203b04cf8872d4649 2023-04-12 02:54:05,461 DEBUG [Listener at localhost/46667] procedure2.Procedure(966): pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_CLOSE_PARENT_REGION; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=315e13e868a7f68a3ce37802b2f6312d, daughterA=4e1092f3c00484f8f09c730861225d43, daughterB=291734c00d212fd203b04cf8872d4649 held the lock before restarting, call acquireLock to restore it. 2023-04-12 02:54:05,461 INFO [Listener at localhost/46667] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_CLOSE_PARENT_REGION; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=315e13e868a7f68a3ce37802b2f6312d, daughterA=4e1092f3c00484f8f09c730861225d43, daughterB=291734c00d212fd203b04cf8872d4649 2023-04-12 02:54:05,461 INFO [Listener at localhost/46667] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_CLOSE_PARENT_REGION; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=315e13e868a7f68a3ce37802b2f6312d, daughterA=4e1092f3c00484f8f09c730861225d43, daughterB=291734c00d212fd203b04cf8872d4649 2023-04-12 02:54:05,461 INFO [Listener at localhost/46667] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_CLOSE_PARENT_REGION; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=315e13e868a7f68a3ce37802b2f6312d, daughterA=4e1092f3c00484f8f09c730861225d43, daughterB=291734c00d212fd203b04cf8872d4649 2023-04-12 02:54:05,461 INFO [Listener at localhost/46667] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 6 msec 2023-04-12 02:54:05,465 DEBUG [Listener at localhost/46667] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-04-12 02:54:05,466 DEBUG [Listener at localhost/46667] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase4.apache.org,44679,1681268031772, table=hbase:meta, region=1588230740 2023-04-12 02:54:05,466 INFO [Listener at localhost/46667] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1681268075466 2023-04-12 02:54:05,468 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_CLOSE_PARENT_REGION; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=315e13e868a7f68a3ce37802b2f6312d, daughterA=4e1092f3c00484f8f09c730861225d43, daughterB=291734c00d212fd203b04cf8872d4649 2023-04-12 02:54:05,468 DEBUG [Listener at localhost/46667] assignment.AssignmentManager(1517): Joining cluster... 2023-04-12 02:54:05,471 DEBUG [Listener at localhost/46667] assignment.RegionStateStore(147): Load hbase:meta entry region=1f255e980a31bb266f2206a30aa2ae3f, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,44679,1681268031772, regionLocation=jenkins-hbase4.apache.org,44679,1681268031772, openSeqNum=2 2023-04-12 02:54:05,471 DEBUG [Listener at localhost/46667] assignment.RegionStateStore(147): Load hbase:meta entry region=315e13e868a7f68a3ce37802b2f6312d, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,34579,1681268031719, regionLocation=jenkins-hbase4.apache.org,34579,1681268031719, openSeqNum=2 2023-04-12 02:54:05,471 INFO [Listener at localhost/46667] assignment.AssignmentManager(1529): Number of RegionServers=3 2023-04-12 02:54:05,471 INFO [Listener at localhost/46667] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1681268105471 2023-04-12 02:54:05,472 INFO [Listener at localhost/46667] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1681268165472 2023-04-12 02:54:05,472 INFO [Listener at localhost/46667] assignment.AssignmentManager(1536): Joined the cluster in 3 msec 2023-04-12 02:54:05,472 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitWithoutPONR, region=315e13e868a7f68a3ce37802b2f6312d, UNASSIGN}] 2023-04-12 02:54:05,472 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-04-12 02:54:05,475 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitWithoutPONR, region=315e13e868a7f68a3ce37802b2f6312d, UNASSIGN 2023-04-12 02:54:05,476 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=29 updating hbase:meta row=315e13e868a7f68a3ce37802b2f6312d, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:05,476 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitWithoutPONR,,1681268041279.315e13e868a7f68a3ce37802b2f6312d.","families":{"info":[{"qualifier":"regioninfo","vlen":54,"tag":[],"timestamp":"1681268045476"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681268045476"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681268045476"}]},"ts":"1681268045476"} 2023-04-12 02:54:05,478 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=30, ppid=29, state=RUNNABLE; CloseRegionProcedure 315e13e868a7f68a3ce37802b2f6312d, server=jenkins-hbase4.apache.org,34579,1681268031719}] 2023-04-12 02:54:05,478 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-04-12 02:54:05,478 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=29, ppid=28, state=WAITING:REGION_STATE_TRANSITION_CONFIRM_CLOSED, locked=true; TransitRegionStateProcedure table=testSplitWithoutPONR, region=315e13e868a7f68a3ce37802b2f6312d, UNASSIGN 2023-04-12 02:54:05,478 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-04-12 02:54:05,478 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=29, ppid=28, state=WAITING:REGION_STATE_TRANSITION_CONFIRM_CLOSED, locked=true; TransitRegionStateProcedure table=testSplitWithoutPONR, region=315e13e868a7f68a3ce37802b2f6312d, UNASSIGN at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-04-12 02:54:05,722 INFO [Listener at localhost/46667] procedure.MasterProcedureTestingUtility(417): Restart 11 exec state=pid=28, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS, locked=true; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=315e13e868a7f68a3ce37802b2f6312d, daughterA=4e1092f3c00484f8f09c730861225d43, daughterB=291734c00d212fd203b04cf8872d4649 2023-04-12 02:54:05,722 INFO [Listener at localhost/46667] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-04-12 02:54:05,722 INFO [Listener at localhost/46667] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-04-12 02:54:05,722 INFO [Listener at localhost/46667] assignment.AssignmentManager(315): Stopping assignment manager 2023-04-12 02:54:05,723 ERROR [Listener at localhost/46667] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-04-12 02:54:05,723 INFO [Listener at localhost/46667] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-04-12 02:54:05,724 INFO [Listener at localhost/46667] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-04-12 02:54:05,724 INFO [Listener at localhost/46667] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-04-12 02:54:05,724 INFO [Listener at localhost/46667] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-04-12 02:54:05,725 INFO [Listener at localhost/46667] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-04-12 02:54:05,726 DEBUG [Listener at localhost/46667] store.ProcedureTree(114): Procedure Procedure(pid=28, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[0, 1, 2] 2023-04-12 02:54:05,726 DEBUG [Listener at localhost/46667] store.ProcedureTree(114): Procedure Procedure(pid=29, ppid=28, class=org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure) stack ids=[] 2023-04-12 02:54:05,727 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-04-12 02:54:05,727 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-04-12 02:54:05,728 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-04-12 02:54:05,728 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-04-12 02:54:05,729 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-12 02:54:05,729 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=cc9012ec0b656baa24d04368fbe026c4, daughterA=e2a9b8fe15a781798c0b49e48ff5aad7, daughterB=6f55b5d6a460bd8c7569dee22f049104 2023-04-12 02:54:05,730 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-12 02:54:05,730 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-12 02:54:05,730 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-04-12 02:54:05,732 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(468): Loading pid=28, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=315e13e868a7f68a3ce37802b2f6312d, daughterA=4e1092f3c00484f8f09c730861225d43, daughterB=291734c00d212fd203b04cf8872d4649 2023-04-12 02:54:05,732 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(468): Loading pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitWithoutPONR, region=315e13e868a7f68a3ce37802b2f6312d, UNASSIGN 2023-04-12 02:54:05,732 DEBUG [Listener at localhost/46667] procedure2.Procedure(966): pid=28, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=315e13e868a7f68a3ce37802b2f6312d, daughterA=4e1092f3c00484f8f09c730861225d43, daughterB=291734c00d212fd203b04cf8872d4649 held the lock before restarting, call acquireLock to restore it. 2023-04-12 02:54:05,732 INFO [Listener at localhost/46667] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=315e13e868a7f68a3ce37802b2f6312d, daughterA=4e1092f3c00484f8f09c730861225d43, daughterB=291734c00d212fd203b04cf8872d4649 2023-04-12 02:54:05,732 INFO [Listener at localhost/46667] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=315e13e868a7f68a3ce37802b2f6312d, daughterA=4e1092f3c00484f8f09c730861225d43, daughterB=291734c00d212fd203b04cf8872d4649 2023-04-12 02:54:05,732 INFO [Listener at localhost/46667] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=315e13e868a7f68a3ce37802b2f6312d, daughterA=4e1092f3c00484f8f09c730861225d43, daughterB=291734c00d212fd203b04cf8872d4649 2023-04-12 02:54:05,733 DEBUG [Listener at localhost/46667] procedure2.Procedure(966): pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitWithoutPONR, region=315e13e868a7f68a3ce37802b2f6312d, UNASSIGN held the lock before restarting, call acquireLock to restore it. 2023-04-12 02:54:05,733 INFO [Listener at localhost/46667] procedure.MasterProcedureScheduler(727): Took xlock for pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitWithoutPONR, region=315e13e868a7f68a3ce37802b2f6312d, UNASSIGN 2023-04-12 02:54:05,733 INFO [Listener at localhost/46667] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 8 msec 2023-04-12 02:54:05,734 DEBUG [Listener at localhost/46667] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-04-12 02:54:05,735 DEBUG [Listener at localhost/46667] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase4.apache.org,44679,1681268031772, table=hbase:meta, region=1588230740 2023-04-12 02:54:05,735 INFO [Listener at localhost/46667] assignment.AssignmentManager(305): Attach pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitWithoutPONR, region=315e13e868a7f68a3ce37802b2f6312d, UNASSIGN to state=OFFLINE, location=null, table=testSplitWithoutPONR, region=315e13e868a7f68a3ce37802b2f6312d to restore RIT 2023-04-12 02:54:05,737 INFO [Listener at localhost/46667] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1681268075737 2023-04-12 02:54:05,738 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitWithoutPONR, region=315e13e868a7f68a3ce37802b2f6312d, UNASSIGN 2023-04-12 02:54:05,738 DEBUG [Listener at localhost/46667] assignment.AssignmentManager(1517): Joining cluster... 2023-04-12 02:54:05,741 DEBUG [Listener at localhost/46667] assignment.RegionStateStore(147): Load hbase:meta entry region=1f255e980a31bb266f2206a30aa2ae3f, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,44679,1681268031772, regionLocation=jenkins-hbase4.apache.org,44679,1681268031772, openSeqNum=2 2023-04-12 02:54:05,742 DEBUG [Listener at localhost/46667] assignment.RegionStateStore(147): Load hbase:meta entry region=315e13e868a7f68a3ce37802b2f6312d, regionState=CLOSING, lastHost=jenkins-hbase4.apache.org,34579,1681268031719, regionLocation=jenkins-hbase4.apache.org,34579,1681268031719, openSeqNum=2 2023-04-12 02:54:05,742 INFO [Listener at localhost/46667] assignment.AssignmentManager(1529): Number of RegionServers=3 2023-04-12 02:54:05,742 INFO [Listener at localhost/46667] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1681268105742 2023-04-12 02:54:05,742 INFO [Listener at localhost/46667] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1681268165742 2023-04-12 02:54:05,742 INFO [Listener at localhost/46667] assignment.AssignmentManager(1536): Joined the cluster in 3 msec 2023-04-12 02:54:05,742 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=29 updating hbase:meta row=315e13e868a7f68a3ce37802b2f6312d, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:05,743 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitWithoutPONR,,1681268041279.315e13e868a7f68a3ce37802b2f6312d.","families":{"info":[{"qualifier":"regioninfo","vlen":54,"tag":[],"timestamp":"1681268045742"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681268045742"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681268045742"}]},"ts":"1681268045742"} 2023-04-12 02:54:05,745 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=30, ppid=29, state=RUNNABLE; CloseRegionProcedure 315e13e868a7f68a3ce37802b2f6312d, server=jenkins-hbase4.apache.org,34579,1681268031719}] 2023-04-12 02:54:05,745 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-04-12 02:54:05,899 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 315e13e868a7f68a3ce37802b2f6312d 2023-04-12 02:54:05,900 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 315e13e868a7f68a3ce37802b2f6312d, disabling compactions & flushes 2023-04-12 02:54:05,900 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region testSplitWithoutPONR,,1681268041279.315e13e868a7f68a3ce37802b2f6312d. 2023-04-12 02:54:05,900 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testSplitWithoutPONR,,1681268041279.315e13e868a7f68a3ce37802b2f6312d. 2023-04-12 02:54:05,900 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on testSplitWithoutPONR,,1681268041279.315e13e868a7f68a3ce37802b2f6312d. after waiting 0 ms 2023-04-12 02:54:05,900 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region testSplitWithoutPONR,,1681268041279.315e13e868a7f68a3ce37802b2f6312d. 2023-04-12 02:54:05,900 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 315e13e868a7f68a3ce37802b2f6312d 2/2 column families, dataSize=480 B heapSize=2.13 KB 2023-04-12 02:54:05,944 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=240 B at sequenceid=82 (bloomFilter=true), to=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/.tmp/cf1/0deaf4065ec441859d88faeb52cfa266 2023-04-12 02:54:06,026 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=240 B at sequenceid=82 (bloomFilter=true), to=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/.tmp/cf2/d253c9057ec44edbad8b758da6e18d8e 2023-04-12 02:54:06,037 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/.tmp/cf1/0deaf4065ec441859d88faeb52cfa266 as hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf1/0deaf4065ec441859d88faeb52cfa266 2023-04-12 02:54:06,046 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf1/0deaf4065ec441859d88faeb52cfa266, entries=8, sequenceid=82, filesize=5.0 K 2023-04-12 02:54:06,048 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/.tmp/cf2/d253c9057ec44edbad8b758da6e18d8e as hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf2/d253c9057ec44edbad8b758da6e18d8e 2023-04-12 02:54:06,056 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf2/d253c9057ec44edbad8b758da6e18d8e, entries=8, sequenceid=82, filesize=5.0 K 2023-04-12 02:54:06,058 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~480 B/480, heapSize ~2.09 KB/2144, currentSize=0 B/0 for 315e13e868a7f68a3ce37802b2f6312d in 158ms, sequenceid=82, compaction requested=true 2023-04-12 02:54:06,087 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/recovered.edits/85.seqid, newMaxSeqId=85, maxSeqId=1 2023-04-12 02:54:06,088 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro 2023-04-12 02:54:06,089 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed testSplitWithoutPONR,,1681268041279.315e13e868a7f68a3ce37802b2f6312d. 2023-04-12 02:54:06,089 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 315e13e868a7f68a3ce37802b2f6312d: 2023-04-12 02:54:06,093 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 315e13e868a7f68a3ce37802b2f6312d 2023-04-12 02:54:06,093 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=29 updating hbase:meta row=315e13e868a7f68a3ce37802b2f6312d, regionState=CLOSED 2023-04-12 02:54:06,094 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testSplitWithoutPONR,,1681268041279.315e13e868a7f68a3ce37802b2f6312d.","families":{"info":[{"qualifier":"regioninfo","vlen":54,"tag":[],"timestamp":"1681268046093"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681268046093"}]},"ts":"1681268046093"} 2023-04-12 02:54:06,095 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-04-12 02:54:06,096 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=30, ppid=29, state=SUCCESS, locked=true; CloseRegionProcedure 315e13e868a7f68a3ce37802b2f6312d, server=jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:06,096 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-04-12 02:54:06,096 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=30, ppid=29, state=SUCCESS, locked=true; CloseRegionProcedure 315e13e868a7f68a3ce37802b2f6312d, server=jenkins-hbase4.apache.org,34579,1681268031719 at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-04-12 02:54:06,243 INFO [Listener at localhost/46667] procedure.MasterProcedureTestingUtility(417): Restart 11 exec state=pid=28, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=315e13e868a7f68a3ce37802b2f6312d, daughterA=4e1092f3c00484f8f09c730861225d43, daughterB=291734c00d212fd203b04cf8872d4649 2023-04-12 02:54:06,243 INFO [Listener at localhost/46667] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-04-12 02:54:06,243 INFO [Listener at localhost/46667] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-04-12 02:54:06,243 INFO [Listener at localhost/46667] assignment.AssignmentManager(315): Stopping assignment manager 2023-04-12 02:54:06,244 ERROR [Listener at localhost/46667] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-04-12 02:54:06,245 INFO [Listener at localhost/46667] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-04-12 02:54:06,245 INFO [Listener at localhost/46667] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-04-12 02:54:06,245 INFO [Listener at localhost/46667] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-04-12 02:54:06,245 INFO [Listener at localhost/46667] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-04-12 02:54:06,246 INFO [Listener at localhost/46667] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-04-12 02:54:06,247 DEBUG [Listener at localhost/46667] store.ProcedureTree(114): Procedure Procedure(pid=28, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[0, 1, 2] 2023-04-12 02:54:06,247 DEBUG [Listener at localhost/46667] store.ProcedureTree(114): Procedure Procedure(pid=29, ppid=28, class=org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure) stack ids=[3] 2023-04-12 02:54:06,247 DEBUG [Listener at localhost/46667] store.ProcedureTree(114): Procedure Procedure(pid=30, ppid=29, class=org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure) stack ids=[4] 2023-04-12 02:54:06,248 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-04-12 02:54:06,248 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-04-12 02:54:06,248 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-04-12 02:54:06,249 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-04-12 02:54:06,249 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-12 02:54:06,250 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=cc9012ec0b656baa24d04368fbe026c4, daughterA=e2a9b8fe15a781798c0b49e48ff5aad7, daughterB=6f55b5d6a460bd8c7569dee22f049104 2023-04-12 02:54:06,250 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-12 02:54:06,250 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-12 02:54:06,250 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-04-12 02:54:06,251 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(468): Loading pid=28, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=315e13e868a7f68a3ce37802b2f6312d, daughterA=4e1092f3c00484f8f09c730861225d43, daughterB=291734c00d212fd203b04cf8872d4649 2023-04-12 02:54:06,251 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(468): Loading pid=29, ppid=28, state=WAITING:REGION_STATE_TRANSITION_CONFIRM_CLOSED; TransitRegionStateProcedure table=testSplitWithoutPONR, region=315e13e868a7f68a3ce37802b2f6312d, UNASSIGN 2023-04-12 02:54:06,251 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(468): Loading pid=30, ppid=29, state=RUNNABLE; CloseRegionProcedure 315e13e868a7f68a3ce37802b2f6312d, server=jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:06,251 DEBUG [Listener at localhost/46667] procedure2.Procedure(966): pid=28, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=315e13e868a7f68a3ce37802b2f6312d, daughterA=4e1092f3c00484f8f09c730861225d43, daughterB=291734c00d212fd203b04cf8872d4649 held the lock before restarting, call acquireLock to restore it. 2023-04-12 02:54:06,251 INFO [Listener at localhost/46667] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=315e13e868a7f68a3ce37802b2f6312d, daughterA=4e1092f3c00484f8f09c730861225d43, daughterB=291734c00d212fd203b04cf8872d4649 2023-04-12 02:54:06,251 INFO [Listener at localhost/46667] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=315e13e868a7f68a3ce37802b2f6312d, daughterA=4e1092f3c00484f8f09c730861225d43, daughterB=291734c00d212fd203b04cf8872d4649 2023-04-12 02:54:06,252 INFO [Listener at localhost/46667] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=315e13e868a7f68a3ce37802b2f6312d, daughterA=4e1092f3c00484f8f09c730861225d43, daughterB=291734c00d212fd203b04cf8872d4649 2023-04-12 02:54:06,252 DEBUG [Listener at localhost/46667] procedure2.Procedure(966): pid=29, ppid=28, state=WAITING:REGION_STATE_TRANSITION_CONFIRM_CLOSED; TransitRegionStateProcedure table=testSplitWithoutPONR, region=315e13e868a7f68a3ce37802b2f6312d, UNASSIGN held the lock before restarting, call acquireLock to restore it. 2023-04-12 02:54:06,252 INFO [Listener at localhost/46667] procedure.MasterProcedureScheduler(727): Took xlock for pid=29, ppid=28, state=WAITING:REGION_STATE_TRANSITION_CONFIRM_CLOSED; TransitRegionStateProcedure table=testSplitWithoutPONR, region=315e13e868a7f68a3ce37802b2f6312d, UNASSIGN 2023-04-12 02:54:06,252 DEBUG [Listener at localhost/46667] procedure2.Procedure(966): pid=30, ppid=29, state=RUNNABLE; CloseRegionProcedure 315e13e868a7f68a3ce37802b2f6312d, server=jenkins-hbase4.apache.org,34579,1681268031719 held the lock before restarting, call acquireLock to restore it. 2023-04-12 02:54:06,252 INFO [Listener at localhost/46667] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 5 msec 2023-04-12 02:54:06,253 DEBUG [Listener at localhost/46667] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-04-12 02:54:06,253 DEBUG [Listener at localhost/46667] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase4.apache.org,44679,1681268031772, table=hbase:meta, region=1588230740 2023-04-12 02:54:06,253 INFO [Listener at localhost/46667] assignment.AssignmentManager(305): Attach pid=29, ppid=28, state=WAITING:REGION_STATE_TRANSITION_CONFIRM_CLOSED; TransitRegionStateProcedure table=testSplitWithoutPONR, region=315e13e868a7f68a3ce37802b2f6312d, UNASSIGN to state=OFFLINE, location=null, table=testSplitWithoutPONR, region=315e13e868a7f68a3ce37802b2f6312d to restore RIT 2023-04-12 02:54:06,255 INFO [Listener at localhost/46667] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1681268076254 2023-04-12 02:54:06,256 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=30, ppid=29, state=RUNNABLE; CloseRegionProcedure 315e13e868a7f68a3ce37802b2f6312d, server=jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:06,256 DEBUG [Listener at localhost/46667] assignment.AssignmentManager(1517): Joining cluster... 2023-04-12 02:54:06,258 DEBUG [Listener at localhost/46667] assignment.RegionStateStore(147): Load hbase:meta entry region=1f255e980a31bb266f2206a30aa2ae3f, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,44679,1681268031772, regionLocation=jenkins-hbase4.apache.org,44679,1681268031772, openSeqNum=2 2023-04-12 02:54:06,258 DEBUG [Listener at localhost/46667] assignment.RegionStateStore(147): Load hbase:meta entry region=315e13e868a7f68a3ce37802b2f6312d, regionState=CLOSED, lastHost=jenkins-hbase4.apache.org,34579,1681268031719, regionLocation=jenkins-hbase4.apache.org,34579,1681268031719, openSeqNum=2 2023-04-12 02:54:06,258 INFO [Listener at localhost/46667] assignment.AssignmentManager(1529): Number of RegionServers=3 2023-04-12 02:54:06,259 INFO [Listener at localhost/46667] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1681268106259 2023-04-12 02:54:06,259 INFO [Listener at localhost/46667] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1681268166259 2023-04-12 02:54:06,259 INFO [Listener at localhost/46667] assignment.AssignmentManager(1536): Joined the cluster in 3 msec 2023-04-12 02:54:06,259 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=29 updating hbase:meta row=315e13e868a7f68a3ce37802b2f6312d, regionState=CLOSED, regionLocation=jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:06,259 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitWithoutPONR,,1681268041279.315e13e868a7f68a3ce37802b2f6312d.","families":{"info":[{"qualifier":"regioninfo","vlen":54,"tag":[],"timestamp":"1681268046259"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681268046259"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681268046259"}]},"ts":"1681268046259"} 2023-04-12 02:54:06,265 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-04-12 02:54:06,268 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=30, resume processing ppid=29 2023-04-12 02:54:06,268 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=30, ppid=29, state=SUCCESS; CloseRegionProcedure 315e13e868a7f68a3ce37802b2f6312d, server=jenkins-hbase4.apache.org,34579,1681268031719 in 520 msec 2023-04-12 02:54:06,268 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-04-12 02:54:06,268 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=29, ppid=28, state=SUCCESS, locked=true; TransitRegionStateProcedure table=testSplitWithoutPONR, region=315e13e868a7f68a3ce37802b2f6312d, UNASSIGN 2023-04-12 02:54:06,268 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-04-12 02:54:06,268 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=29, ppid=28, state=SUCCESS, locked=true; TransitRegionStateProcedure table=testSplitWithoutPONR, region=315e13e868a7f68a3ce37802b2f6312d, UNASSIGN at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-04-12 02:54:06,509 INFO [Listener at localhost/46667] procedure.MasterProcedureTestingUtility(417): Restart 11 exec state=pid=28, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=315e13e868a7f68a3ce37802b2f6312d, daughterA=4e1092f3c00484f8f09c730861225d43, daughterB=291734c00d212fd203b04cf8872d4649 2023-04-12 02:54:06,509 INFO [Listener at localhost/46667] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-04-12 02:54:06,509 INFO [Listener at localhost/46667] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-04-12 02:54:06,509 INFO [Listener at localhost/46667] assignment.AssignmentManager(315): Stopping assignment manager 2023-04-12 02:54:06,511 ERROR [Listener at localhost/46667] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-04-12 02:54:06,511 INFO [Listener at localhost/46667] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-04-12 02:54:06,511 INFO [Listener at localhost/46667] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-04-12 02:54:06,511 INFO [Listener at localhost/46667] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-04-12 02:54:06,511 INFO [Listener at localhost/46667] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-04-12 02:54:06,512 INFO [Listener at localhost/46667] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-04-12 02:54:06,515 DEBUG [Listener at localhost/46667] store.ProcedureTree(114): Procedure Procedure(pid=28, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[0, 1, 2] 2023-04-12 02:54:06,515 DEBUG [Listener at localhost/46667] store.ProcedureTree(114): Procedure Procedure(pid=29, ppid=28, class=org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure) stack ids=[3] 2023-04-12 02:54:06,515 DEBUG [Listener at localhost/46667] store.ProcedureTree(114): Procedure Procedure(pid=30, ppid=29, class=org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure) stack ids=[4, 5] 2023-04-12 02:54:06,515 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-04-12 02:54:06,516 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-04-12 02:54:06,516 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-04-12 02:54:06,517 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-04-12 02:54:06,517 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-12 02:54:06,518 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=cc9012ec0b656baa24d04368fbe026c4, daughterA=e2a9b8fe15a781798c0b49e48ff5aad7, daughterB=6f55b5d6a460bd8c7569dee22f049104 2023-04-12 02:54:06,518 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-12 02:54:06,518 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-12 02:54:06,519 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-04-12 02:54:06,520 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(468): Loading pid=28, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=315e13e868a7f68a3ce37802b2f6312d, daughterA=4e1092f3c00484f8f09c730861225d43, daughterB=291734c00d212fd203b04cf8872d4649 2023-04-12 02:54:06,520 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(468): Loading pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_CONFIRM_CLOSED; TransitRegionStateProcedure table=testSplitWithoutPONR, region=315e13e868a7f68a3ce37802b2f6312d, UNASSIGN 2023-04-12 02:54:06,520 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(468): Loading pid=30, ppid=29, state=SUCCESS; CloseRegionProcedure 315e13e868a7f68a3ce37802b2f6312d, server=jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:06,520 DEBUG [Listener at localhost/46667] procedure2.Procedure(966): pid=28, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=315e13e868a7f68a3ce37802b2f6312d, daughterA=4e1092f3c00484f8f09c730861225d43, daughterB=291734c00d212fd203b04cf8872d4649 held the lock before restarting, call acquireLock to restore it. 2023-04-12 02:54:06,520 INFO [Listener at localhost/46667] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=315e13e868a7f68a3ce37802b2f6312d, daughterA=4e1092f3c00484f8f09c730861225d43, daughterB=291734c00d212fd203b04cf8872d4649 2023-04-12 02:54:06,520 INFO [Listener at localhost/46667] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=315e13e868a7f68a3ce37802b2f6312d, daughterA=4e1092f3c00484f8f09c730861225d43, daughterB=291734c00d212fd203b04cf8872d4649 2023-04-12 02:54:06,521 INFO [Listener at localhost/46667] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=315e13e868a7f68a3ce37802b2f6312d, daughterA=4e1092f3c00484f8f09c730861225d43, daughterB=291734c00d212fd203b04cf8872d4649 2023-04-12 02:54:06,521 DEBUG [Listener at localhost/46667] procedure2.Procedure(946): pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_CONFIRM_CLOSED; TransitRegionStateProcedure table=testSplitWithoutPONR, region=315e13e868a7f68a3ce37802b2f6312d, UNASSIGN didn't hold the lock before restarting, skip acquiring lock. 2023-04-12 02:54:06,521 DEBUG [Listener at localhost/46667] procedure2.Procedure(946): pid=30, ppid=29, state=SUCCESS; CloseRegionProcedure 315e13e868a7f68a3ce37802b2f6312d, server=jenkins-hbase4.apache.org,34579,1681268031719 didn't hold the lock before restarting, skip acquiring lock. 2023-04-12 02:54:06,521 INFO [Listener at localhost/46667] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 8 msec 2023-04-12 02:54:06,522 DEBUG [Listener at localhost/46667] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-04-12 02:54:06,523 DEBUG [Listener at localhost/46667] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase4.apache.org,44679,1681268031772, table=hbase:meta, region=1588230740 2023-04-12 02:54:06,523 INFO [Listener at localhost/46667] assignment.AssignmentManager(305): Attach pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_CONFIRM_CLOSED; TransitRegionStateProcedure table=testSplitWithoutPONR, region=315e13e868a7f68a3ce37802b2f6312d, UNASSIGN to state=OFFLINE, location=null, table=testSplitWithoutPONR, region=315e13e868a7f68a3ce37802b2f6312d to restore RIT 2023-04-12 02:54:06,523 INFO [Listener at localhost/46667] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1681268076523 2023-04-12 02:54:06,524 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_CONFIRM_CLOSED; TransitRegionStateProcedure table=testSplitWithoutPONR, region=315e13e868a7f68a3ce37802b2f6312d, UNASSIGN 2023-04-12 02:54:06,524 DEBUG [Listener at localhost/46667] assignment.AssignmentManager(1517): Joining cluster... 2023-04-12 02:54:06,528 DEBUG [Listener at localhost/46667] assignment.RegionStateStore(147): Load hbase:meta entry region=1f255e980a31bb266f2206a30aa2ae3f, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,44679,1681268031772, regionLocation=jenkins-hbase4.apache.org,44679,1681268031772, openSeqNum=2 2023-04-12 02:54:06,528 DEBUG [Listener at localhost/46667] assignment.RegionStateStore(147): Load hbase:meta entry region=315e13e868a7f68a3ce37802b2f6312d, regionState=CLOSED, lastHost=jenkins-hbase4.apache.org,34579,1681268031719, regionLocation=jenkins-hbase4.apache.org,34579,1681268031719, openSeqNum=2 2023-04-12 02:54:06,528 INFO [Listener at localhost/46667] assignment.AssignmentManager(1529): Number of RegionServers=3 2023-04-12 02:54:06,528 INFO [Listener at localhost/46667] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1681268106528 2023-04-12 02:54:06,528 INFO [Listener at localhost/46667] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1681268166528 2023-04-12 02:54:06,529 INFO [Listener at localhost/46667] assignment.AssignmentManager(1536): Joined the cluster in 4 msec 2023-04-12 02:54:06,529 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_CONFIRM_CLOSED; TransitRegionStateProcedure table=testSplitWithoutPONR, region=315e13e868a7f68a3ce37802b2f6312d, UNASSIGN 2023-04-12 02:54:06,531 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-04-12 02:54:06,539 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=29, resume processing ppid=28 2023-04-12 02:54:06,539 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=29, ppid=28, state=SUCCESS; TransitRegionStateProcedure table=testSplitWithoutPONR, region=315e13e868a7f68a3ce37802b2f6312d, UNASSIGN in 1.0590 sec 2023-04-12 02:54:06,546 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-04-12 02:54:06,546 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_CREATE_DAUGHTER_REGIONS, locked=true; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=315e13e868a7f68a3ce37802b2f6312d, daughterA=4e1092f3c00484f8f09c730861225d43, daughterB=291734c00d212fd203b04cf8872d4649 2023-04-12 02:54:06,546 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-04-12 02:54:06,546 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_CREATE_DAUGHTER_REGIONS, locked=true; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=315e13e868a7f68a3ce37802b2f6312d, daughterA=4e1092f3c00484f8f09c730861225d43, daughterB=291734c00d212fd203b04cf8872d4649 at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-04-12 02:54:06,779 INFO [Listener at localhost/46667] procedure.MasterProcedureTestingUtility(417): Restart 4 exec state=pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_CREATE_DAUGHTER_REGIONS, locked=true; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=315e13e868a7f68a3ce37802b2f6312d, daughterA=4e1092f3c00484f8f09c730861225d43, daughterB=291734c00d212fd203b04cf8872d4649 2023-04-12 02:54:06,779 INFO [Listener at localhost/46667] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-04-12 02:54:06,779 INFO [Listener at localhost/46667] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-04-12 02:54:06,779 INFO [Listener at localhost/46667] assignment.AssignmentManager(315): Stopping assignment manager 2023-04-12 02:54:06,780 ERROR [Listener at localhost/46667] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-04-12 02:54:06,781 INFO [Listener at localhost/46667] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-04-12 02:54:06,781 INFO [Listener at localhost/46667] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-04-12 02:54:06,781 INFO [Listener at localhost/46667] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-04-12 02:54:06,781 INFO [Listener at localhost/46667] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-04-12 02:54:06,782 INFO [Listener at localhost/46667] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-04-12 02:54:06,785 DEBUG [Listener at localhost/46667] store.ProcedureTree(114): Procedure Procedure(pid=28, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[0, 1, 2] 2023-04-12 02:54:06,785 DEBUG [Listener at localhost/46667] store.ProcedureTree(114): Procedure Procedure(pid=29, ppid=28, class=org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure) stack ids=[3, 6] 2023-04-12 02:54:06,785 DEBUG [Listener at localhost/46667] store.ProcedureTree(114): Procedure Procedure(pid=30, ppid=29, class=org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure) stack ids=[4, 5] 2023-04-12 02:54:06,786 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-04-12 02:54:06,786 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-04-12 02:54:06,786 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-04-12 02:54:06,786 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-04-12 02:54:06,787 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-12 02:54:06,787 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=cc9012ec0b656baa24d04368fbe026c4, daughterA=e2a9b8fe15a781798c0b49e48ff5aad7, daughterB=6f55b5d6a460bd8c7569dee22f049104 2023-04-12 02:54:06,788 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-12 02:54:06,788 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-12 02:54:06,789 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-04-12 02:54:06,790 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(468): Loading pid=28, state=RUNNABLE:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=315e13e868a7f68a3ce37802b2f6312d, daughterA=4e1092f3c00484f8f09c730861225d43, daughterB=291734c00d212fd203b04cf8872d4649 2023-04-12 02:54:06,790 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(468): Loading pid=29, ppid=28, state=SUCCESS; TransitRegionStateProcedure table=testSplitWithoutPONR, region=315e13e868a7f68a3ce37802b2f6312d, UNASSIGN 2023-04-12 02:54:06,790 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(468): Loading pid=30, ppid=29, state=SUCCESS; CloseRegionProcedure 315e13e868a7f68a3ce37802b2f6312d, server=jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:06,790 DEBUG [Listener at localhost/46667] procedure2.Procedure(946): pid=28, state=RUNNABLE:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=315e13e868a7f68a3ce37802b2f6312d, daughterA=4e1092f3c00484f8f09c730861225d43, daughterB=291734c00d212fd203b04cf8872d4649 didn't hold the lock before restarting, skip acquiring lock. 2023-04-12 02:54:06,790 DEBUG [Listener at localhost/46667] procedure2.Procedure(946): pid=29, ppid=28, state=SUCCESS; TransitRegionStateProcedure table=testSplitWithoutPONR, region=315e13e868a7f68a3ce37802b2f6312d, UNASSIGN didn't hold the lock before restarting, skip acquiring lock. 2023-04-12 02:54:06,790 DEBUG [Listener at localhost/46667] procedure2.Procedure(946): pid=30, ppid=29, state=SUCCESS; CloseRegionProcedure 315e13e868a7f68a3ce37802b2f6312d, server=jenkins-hbase4.apache.org,34579,1681268031719 didn't hold the lock before restarting, skip acquiring lock. 2023-04-12 02:54:06,790 INFO [Listener at localhost/46667] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 8 msec 2023-04-12 02:54:06,791 DEBUG [Listener at localhost/46667] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-04-12 02:54:06,792 DEBUG [Listener at localhost/46667] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase4.apache.org,44679,1681268031772, table=hbase:meta, region=1588230740 2023-04-12 02:54:06,800 INFO [Listener at localhost/46667] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1681268076800 2023-04-12 02:54:06,803 DEBUG [Listener at localhost/46667] assignment.AssignmentManager(1517): Joining cluster... 2023-04-12 02:54:06,803 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=28, state=RUNNABLE:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=315e13e868a7f68a3ce37802b2f6312d, daughterA=4e1092f3c00484f8f09c730861225d43, daughterB=291734c00d212fd203b04cf8872d4649 2023-04-12 02:54:06,806 DEBUG [Listener at localhost/46667] assignment.RegionStateStore(147): Load hbase:meta entry region=1f255e980a31bb266f2206a30aa2ae3f, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,44679,1681268031772, regionLocation=jenkins-hbase4.apache.org,44679,1681268031772, openSeqNum=2 2023-04-12 02:54:06,807 DEBUG [Listener at localhost/46667] assignment.RegionStateStore(147): Load hbase:meta entry region=315e13e868a7f68a3ce37802b2f6312d, regionState=CLOSED, lastHost=jenkins-hbase4.apache.org,34579,1681268031719, regionLocation=jenkins-hbase4.apache.org,34579,1681268031719, openSeqNum=2 2023-04-12 02:54:06,807 INFO [Listener at localhost/46667] assignment.AssignmentManager(1529): Number of RegionServers=3 2023-04-12 02:54:06,807 INFO [Listener at localhost/46667] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1681268106807 2023-04-12 02:54:06,807 INFO [Listener at localhost/46667] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1681268166807 2023-04-12 02:54:06,807 INFO [Listener at localhost/46667] assignment.AssignmentManager(1536): Joined the cluster in 4 msec 2023-04-12 02:54:06,808 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, state=RUNNABLE:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=315e13e868a7f68a3ce37802b2f6312d, daughterA=4e1092f3c00484f8f09c730861225d43, daughterB=291734c00d212fd203b04cf8872d4649 2023-04-12 02:54:06,808 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, state=RUNNABLE:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=315e13e868a7f68a3ce37802b2f6312d, daughterA=4e1092f3c00484f8f09c730861225d43, daughterB=291734c00d212fd203b04cf8872d4649 2023-04-12 02:54:06,808 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, state=RUNNABLE:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=315e13e868a7f68a3ce37802b2f6312d, daughterA=4e1092f3c00484f8f09c730861225d43, daughterB=291734c00d212fd203b04cf8872d4649 2023-04-12 02:54:06,816 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-04-12 02:54:06,833 INFO [PEWorker-1] assignment.SplitTableRegionProcedure(694): pid=28 splitting 14 storefiles, region=315e13e868a7f68a3ce37802b2f6312d, threads=14 2023-04-12 02:54:06,847 DEBUG [StoreFileSplitter-pool-0] assignment.SplitTableRegionProcedure(776): pid=28 splitting started for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf2/0b28a4ef23e442f38cfdf4760e6f9480 for region: 315e13e868a7f68a3ce37802b2f6312d 2023-04-12 02:54:06,859 DEBUG [StoreFileSplitter-pool-1] assignment.SplitTableRegionProcedure(776): pid=28 splitting started for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf2/2540a68ffbdf4440b5ff30ca6a36fada for region: 315e13e868a7f68a3ce37802b2f6312d 2023-04-12 02:54:06,859 DEBUG [StoreFileSplitter-pool-3] assignment.SplitTableRegionProcedure(776): pid=28 splitting started for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf2/babf83b74cec452290ff19cd45eb4713 for region: 315e13e868a7f68a3ce37802b2f6312d 2023-04-12 02:54:06,860 DEBUG [StoreFileSplitter-pool-4] assignment.SplitTableRegionProcedure(776): pid=28 splitting started for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf2/d253c9057ec44edbad8b758da6e18d8e for region: 315e13e868a7f68a3ce37802b2f6312d 2023-04-12 02:54:06,867 DEBUG [StoreFileSplitter-pool-6] assignment.SplitTableRegionProcedure(776): pid=28 splitting started for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf2/f13ea6a8012d44709becae664369d2d0 for region: 315e13e868a7f68a3ce37802b2f6312d 2023-04-12 02:54:06,868 DEBUG [StoreFileSplitter-pool-7] assignment.SplitTableRegionProcedure(776): pid=28 splitting started for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf1/0deaf4065ec441859d88faeb52cfa266 for region: 315e13e868a7f68a3ce37802b2f6312d 2023-04-12 02:54:06,869 DEBUG [StoreFileSplitter-pool-8] assignment.SplitTableRegionProcedure(776): pid=28 splitting started for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf1/1cabdcde78c14f6a8ec70790c1ae4cc4 for region: 315e13e868a7f68a3ce37802b2f6312d 2023-04-12 02:54:06,861 DEBUG [StoreFileSplitter-pool-5] assignment.SplitTableRegionProcedure(776): pid=28 splitting started for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf2/d5ab6ea45d1b4298ae7d38df0de5a43f for region: 315e13e868a7f68a3ce37802b2f6312d 2023-04-12 02:54:06,869 DEBUG [StoreFileSplitter-pool-2] assignment.SplitTableRegionProcedure(776): pid=28 splitting started for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf2/3a80c419c6b94682b44cf75bc8aef7bd for region: 315e13e868a7f68a3ce37802b2f6312d 2023-04-12 02:54:06,870 DEBUG [StoreFileSplitter-pool-9] assignment.SplitTableRegionProcedure(776): pid=28 splitting started for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf1/2ca08ea408834a75bc73ef0d3d78fa73 for region: 315e13e868a7f68a3ce37802b2f6312d 2023-04-12 02:54:06,876 DEBUG [StoreFileSplitter-pool-10] assignment.SplitTableRegionProcedure(776): pid=28 splitting started for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf1/2fd181776c3749c18358aebdfc597014 for region: 315e13e868a7f68a3ce37802b2f6312d 2023-04-12 02:54:06,880 DEBUG [StoreFileSplitter-pool-11] assignment.SplitTableRegionProcedure(776): pid=28 splitting started for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf1/55bdd10dae634a008e897b8e86048f3c for region: 315e13e868a7f68a3ce37802b2f6312d 2023-04-12 02:54:06,880 DEBUG [StoreFileSplitter-pool-12] assignment.SplitTableRegionProcedure(776): pid=28 splitting started for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf1/713becfd985e40a8892fbc1f6fe6bdb4 for region: 315e13e868a7f68a3ce37802b2f6312d 2023-04-12 02:54:06,888 DEBUG [StoreFileSplitter-pool-13] assignment.SplitTableRegionProcedure(776): pid=28 splitting started for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf1/7e0f8c9c61e247ecbfc4c27206fae340 for region: 315e13e868a7f68a3ce37802b2f6312d 2023-04-12 02:54:07,415 DEBUG [StoreFileSplitter-pool-0] assignment.SplitTableRegionProcedure(787): pid=28 splitting complete for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf2/0b28a4ef23e442f38cfdf4760e6f9480 for region: 315e13e868a7f68a3ce37802b2f6312d 2023-04-12 02:54:07,416 DEBUG [StoreFileSplitter-pool-9] assignment.SplitTableRegionProcedure(787): pid=28 splitting complete for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf1/2ca08ea408834a75bc73ef0d3d78fa73 for region: 315e13e868a7f68a3ce37802b2f6312d 2023-04-12 02:54:07,428 DEBUG [StoreFileSplitter-pool-10] assignment.SplitTableRegionProcedure(787): pid=28 splitting complete for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf1/2fd181776c3749c18358aebdfc597014 for region: 315e13e868a7f68a3ce37802b2f6312d 2023-04-12 02:54:07,436 DEBUG [StoreFileSplitter-pool-3] assignment.SplitTableRegionProcedure(787): pid=28 splitting complete for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf2/babf83b74cec452290ff19cd45eb4713 for region: 315e13e868a7f68a3ce37802b2f6312d 2023-04-12 02:54:07,462 DEBUG [StoreFileSplitter-pool-1] assignment.SplitTableRegionProcedure(787): pid=28 splitting complete for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf2/2540a68ffbdf4440b5ff30ca6a36fada for region: 315e13e868a7f68a3ce37802b2f6312d 2023-04-12 02:54:07,464 DEBUG [StoreFileSplitter-pool-7] assignment.SplitTableRegionProcedure(787): pid=28 splitting complete for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf1/0deaf4065ec441859d88faeb52cfa266 for region: 315e13e868a7f68a3ce37802b2f6312d 2023-04-12 02:54:07,465 DEBUG [StoreFileSplitter-pool-2] assignment.SplitTableRegionProcedure(787): pid=28 splitting complete for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf2/3a80c419c6b94682b44cf75bc8aef7bd for region: 315e13e868a7f68a3ce37802b2f6312d 2023-04-12 02:54:07,496 DEBUG [StoreFileSplitter-pool-12] assignment.SplitTableRegionProcedure(787): pid=28 splitting complete for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf1/713becfd985e40a8892fbc1f6fe6bdb4 for region: 315e13e868a7f68a3ce37802b2f6312d 2023-04-12 02:54:07,499 DEBUG [StoreFileSplitter-pool-5] assignment.SplitTableRegionProcedure(787): pid=28 splitting complete for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf2/d5ab6ea45d1b4298ae7d38df0de5a43f for region: 315e13e868a7f68a3ce37802b2f6312d 2023-04-12 02:54:07,503 DEBUG [StoreFileSplitter-pool-13] assignment.SplitTableRegionProcedure(787): pid=28 splitting complete for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf1/7e0f8c9c61e247ecbfc4c27206fae340 for region: 315e13e868a7f68a3ce37802b2f6312d 2023-04-12 02:54:07,720 DEBUG [StoreFileSplitter-pool-4] assignment.SplitTableRegionProcedure(787): pid=28 splitting complete for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf2/d253c9057ec44edbad8b758da6e18d8e for region: 315e13e868a7f68a3ce37802b2f6312d 2023-04-12 02:54:07,747 DEBUG [StoreFileSplitter-pool-11] assignment.SplitTableRegionProcedure(787): pid=28 splitting complete for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf1/55bdd10dae634a008e897b8e86048f3c for region: 315e13e868a7f68a3ce37802b2f6312d 2023-04-12 02:54:07,766 DEBUG [StoreFileSplitter-pool-8] assignment.SplitTableRegionProcedure(787): pid=28 splitting complete for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf1/1cabdcde78c14f6a8ec70790c1ae4cc4 for region: 315e13e868a7f68a3ce37802b2f6312d 2023-04-12 02:54:07,819 DEBUG [StoreFileSplitter-pool-6] assignment.SplitTableRegionProcedure(787): pid=28 splitting complete for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf2/f13ea6a8012d44709becae664369d2d0 for region: 315e13e868a7f68a3ce37802b2f6312d 2023-04-12 02:54:07,820 DEBUG [PEWorker-1] assignment.SplitTableRegionProcedure(755): pid=28 split storefiles for region 315e13e868a7f68a3ce37802b2f6312d Daughter A: 14 storefiles, Daughter B: 14 storefiles. 2023-04-12 02:54:07,944 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-04-12 02:54:07,944 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_WRITE_MAX_SEQUENCE_ID_FILE, locked=true; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=315e13e868a7f68a3ce37802b2f6312d, daughterA=4e1092f3c00484f8f09c730861225d43, daughterB=291734c00d212fd203b04cf8872d4649 2023-04-12 02:54:07,944 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-04-12 02:54:07,944 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_WRITE_MAX_SEQUENCE_ID_FILE, locked=true; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=315e13e868a7f68a3ce37802b2f6312d, daughterA=4e1092f3c00484f8f09c730861225d43, daughterB=291734c00d212fd203b04cf8872d4649 at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-04-12 02:54:08,058 INFO [Listener at localhost/46667] procedure.MasterProcedureTestingUtility(417): Restart 5 exec state=pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_WRITE_MAX_SEQUENCE_ID_FILE, locked=true; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=315e13e868a7f68a3ce37802b2f6312d, daughterA=4e1092f3c00484f8f09c730861225d43, daughterB=291734c00d212fd203b04cf8872d4649 2023-04-12 02:54:08,058 INFO [Listener at localhost/46667] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-04-12 02:54:08,058 INFO [Listener at localhost/46667] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-04-12 02:54:08,059 INFO [Listener at localhost/46667] assignment.AssignmentManager(315): Stopping assignment manager 2023-04-12 02:54:08,060 ERROR [Listener at localhost/46667] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-04-12 02:54:08,061 INFO [Listener at localhost/46667] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-04-12 02:54:08,061 INFO [Listener at localhost/46667] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-04-12 02:54:08,061 INFO [Listener at localhost/46667] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-04-12 02:54:08,061 INFO [Listener at localhost/46667] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-04-12 02:54:08,062 INFO [Listener at localhost/46667] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-04-12 02:54:08,063 DEBUG [Listener at localhost/46667] store.ProcedureTree(114): Procedure Procedure(pid=28, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[0, 1, 2, 7] 2023-04-12 02:54:08,064 DEBUG [Listener at localhost/46667] store.ProcedureTree(114): Procedure Procedure(pid=29, ppid=28, class=org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure) stack ids=[3, 6] 2023-04-12 02:54:08,064 DEBUG [Listener at localhost/46667] store.ProcedureTree(114): Procedure Procedure(pid=30, ppid=29, class=org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure) stack ids=[4, 5] 2023-04-12 02:54:08,064 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-04-12 02:54:08,065 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-04-12 02:54:08,065 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-04-12 02:54:08,066 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-04-12 02:54:08,066 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-12 02:54:08,067 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=cc9012ec0b656baa24d04368fbe026c4, daughterA=e2a9b8fe15a781798c0b49e48ff5aad7, daughterB=6f55b5d6a460bd8c7569dee22f049104 2023-04-12 02:54:08,068 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-12 02:54:08,068 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-12 02:54:08,069 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-04-12 02:54:08,070 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(468): Loading pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_CREATE_DAUGHTER_REGIONS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=315e13e868a7f68a3ce37802b2f6312d, daughterA=4e1092f3c00484f8f09c730861225d43, daughterB=291734c00d212fd203b04cf8872d4649 2023-04-12 02:54:08,070 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(468): Loading pid=29, ppid=28, state=SUCCESS; TransitRegionStateProcedure table=testSplitWithoutPONR, region=315e13e868a7f68a3ce37802b2f6312d, UNASSIGN 2023-04-12 02:54:08,070 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(468): Loading pid=30, ppid=29, state=SUCCESS; CloseRegionProcedure 315e13e868a7f68a3ce37802b2f6312d, server=jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:08,071 DEBUG [Listener at localhost/46667] procedure2.Procedure(966): pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_CREATE_DAUGHTER_REGIONS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=315e13e868a7f68a3ce37802b2f6312d, daughterA=4e1092f3c00484f8f09c730861225d43, daughterB=291734c00d212fd203b04cf8872d4649 held the lock before restarting, call acquireLock to restore it. 2023-04-12 02:54:08,071 INFO [Listener at localhost/46667] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_CREATE_DAUGHTER_REGIONS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=315e13e868a7f68a3ce37802b2f6312d, daughterA=4e1092f3c00484f8f09c730861225d43, daughterB=291734c00d212fd203b04cf8872d4649 2023-04-12 02:54:08,071 INFO [Listener at localhost/46667] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_CREATE_DAUGHTER_REGIONS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=315e13e868a7f68a3ce37802b2f6312d, daughterA=4e1092f3c00484f8f09c730861225d43, daughterB=291734c00d212fd203b04cf8872d4649 2023-04-12 02:54:08,071 INFO [Listener at localhost/46667] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_CREATE_DAUGHTER_REGIONS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=315e13e868a7f68a3ce37802b2f6312d, daughterA=4e1092f3c00484f8f09c730861225d43, daughterB=291734c00d212fd203b04cf8872d4649 2023-04-12 02:54:08,071 DEBUG [Listener at localhost/46667] procedure2.Procedure(946): pid=29, ppid=28, state=SUCCESS; TransitRegionStateProcedure table=testSplitWithoutPONR, region=315e13e868a7f68a3ce37802b2f6312d, UNASSIGN didn't hold the lock before restarting, skip acquiring lock. 2023-04-12 02:54:08,071 DEBUG [Listener at localhost/46667] procedure2.Procedure(946): pid=30, ppid=29, state=SUCCESS; CloseRegionProcedure 315e13e868a7f68a3ce37802b2f6312d, server=jenkins-hbase4.apache.org,34579,1681268031719 didn't hold the lock before restarting, skip acquiring lock. 2023-04-12 02:54:08,071 INFO [Listener at localhost/46667] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 9 msec 2023-04-12 02:54:08,072 DEBUG [Listener at localhost/46667] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-04-12 02:54:08,073 DEBUG [Listener at localhost/46667] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase4.apache.org,44679,1681268031772, table=hbase:meta, region=1588230740 2023-04-12 02:54:08,074 INFO [Listener at localhost/46667] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1681268078074 2023-04-12 02:54:08,075 DEBUG [Listener at localhost/46667] assignment.AssignmentManager(1517): Joining cluster... 2023-04-12 02:54:08,079 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_CREATE_DAUGHTER_REGIONS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=315e13e868a7f68a3ce37802b2f6312d, daughterA=4e1092f3c00484f8f09c730861225d43, daughterB=291734c00d212fd203b04cf8872d4649 2023-04-12 02:54:08,081 DEBUG [Listener at localhost/46667] assignment.RegionStateStore(147): Load hbase:meta entry region=1f255e980a31bb266f2206a30aa2ae3f, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,44679,1681268031772, regionLocation=jenkins-hbase4.apache.org,44679,1681268031772, openSeqNum=2 2023-04-12 02:54:08,081 DEBUG [Listener at localhost/46667] assignment.RegionStateStore(147): Load hbase:meta entry region=315e13e868a7f68a3ce37802b2f6312d, regionState=CLOSED, lastHost=jenkins-hbase4.apache.org,34579,1681268031719, regionLocation=jenkins-hbase4.apache.org,34579,1681268031719, openSeqNum=2 2023-04-12 02:54:08,081 INFO [Listener at localhost/46667] assignment.AssignmentManager(1529): Number of RegionServers=3 2023-04-12 02:54:08,081 INFO [Listener at localhost/46667] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1681268108081 2023-04-12 02:54:08,081 INFO [Listener at localhost/46667] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1681268168081 2023-04-12 02:54:08,081 INFO [Listener at localhost/46667] assignment.AssignmentManager(1536): Joined the cluster in 6 msec 2023-04-12 02:54:08,101 INFO [PEWorker-1] assignment.SplitTableRegionProcedure(694): pid=28 splitting 14 storefiles, region=315e13e868a7f68a3ce37802b2f6312d, threads=14 2023-04-12 02:54:08,103 DEBUG [StoreFileSplitter-pool-0] assignment.SplitTableRegionProcedure(776): pid=28 splitting started for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf2/0b28a4ef23e442f38cfdf4760e6f9480 for region: 315e13e868a7f68a3ce37802b2f6312d 2023-04-12 02:54:08,103 DEBUG [StoreFileSplitter-pool-1] assignment.SplitTableRegionProcedure(776): pid=28 splitting started for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf2/2540a68ffbdf4440b5ff30ca6a36fada for region: 315e13e868a7f68a3ce37802b2f6312d 2023-04-12 02:54:08,104 DEBUG [StoreFileSplitter-pool-2] assignment.SplitTableRegionProcedure(776): pid=28 splitting started for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf2/3a80c419c6b94682b44cf75bc8aef7bd for region: 315e13e868a7f68a3ce37802b2f6312d 2023-04-12 02:54:08,113 DEBUG [StoreFileSplitter-pool-3] assignment.SplitTableRegionProcedure(776): pid=28 splitting started for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf2/babf83b74cec452290ff19cd45eb4713 for region: 315e13e868a7f68a3ce37802b2f6312d 2023-04-12 02:54:08,113 DEBUG [StoreFileSplitter-pool-4] assignment.SplitTableRegionProcedure(776): pid=28 splitting started for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf2/d253c9057ec44edbad8b758da6e18d8e for region: 315e13e868a7f68a3ce37802b2f6312d 2023-04-12 02:54:08,114 DEBUG [StoreFileSplitter-pool-5] assignment.SplitTableRegionProcedure(776): pid=28 splitting started for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf2/d5ab6ea45d1b4298ae7d38df0de5a43f for region: 315e13e868a7f68a3ce37802b2f6312d 2023-04-12 02:54:08,116 DEBUG [StoreFileSplitter-pool-6] assignment.SplitTableRegionProcedure(776): pid=28 splitting started for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf2/f13ea6a8012d44709becae664369d2d0 for region: 315e13e868a7f68a3ce37802b2f6312d 2023-04-12 02:54:08,119 DEBUG [StoreFileSplitter-pool-7] assignment.SplitTableRegionProcedure(776): pid=28 splitting started for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf1/0deaf4065ec441859d88faeb52cfa266 for region: 315e13e868a7f68a3ce37802b2f6312d 2023-04-12 02:54:08,124 DEBUG [StoreFileSplitter-pool-8] assignment.SplitTableRegionProcedure(776): pid=28 splitting started for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf1/1cabdcde78c14f6a8ec70790c1ae4cc4 for region: 315e13e868a7f68a3ce37802b2f6312d 2023-04-12 02:54:08,128 DEBUG [StoreFileSplitter-pool-9] assignment.SplitTableRegionProcedure(776): pid=28 splitting started for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf1/2ca08ea408834a75bc73ef0d3d78fa73 for region: 315e13e868a7f68a3ce37802b2f6312d 2023-04-12 02:54:08,132 DEBUG [StoreFileSplitter-pool-10] assignment.SplitTableRegionProcedure(776): pid=28 splitting started for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf1/2fd181776c3749c18358aebdfc597014 for region: 315e13e868a7f68a3ce37802b2f6312d 2023-04-12 02:54:08,145 DEBUG [StoreFileSplitter-pool-11] assignment.SplitTableRegionProcedure(776): pid=28 splitting started for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf1/55bdd10dae634a008e897b8e86048f3c for region: 315e13e868a7f68a3ce37802b2f6312d 2023-04-12 02:54:08,156 DEBUG [StoreFileSplitter-pool-12] assignment.SplitTableRegionProcedure(776): pid=28 splitting started for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf1/713becfd985e40a8892fbc1f6fe6bdb4 for region: 315e13e868a7f68a3ce37802b2f6312d 2023-04-12 02:54:08,168 DEBUG [StoreFileSplitter-pool-13] assignment.SplitTableRegionProcedure(776): pid=28 splitting started for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf1/7e0f8c9c61e247ecbfc4c27206fae340 for region: 315e13e868a7f68a3ce37802b2f6312d 2023-04-12 02:54:08,460 DEBUG [StoreFileSplitter-pool-0] assignment.SplitTableRegionProcedure(787): pid=28 splitting complete for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf2/0b28a4ef23e442f38cfdf4760e6f9480 for region: 315e13e868a7f68a3ce37802b2f6312d 2023-04-12 02:54:08,684 DEBUG [StoreFileSplitter-pool-3] assignment.SplitTableRegionProcedure(787): pid=28 splitting complete for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf2/babf83b74cec452290ff19cd45eb4713 for region: 315e13e868a7f68a3ce37802b2f6312d 2023-04-12 02:54:08,766 DEBUG [StoreFileSplitter-pool-1] assignment.SplitTableRegionProcedure(787): pid=28 splitting complete for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf2/2540a68ffbdf4440b5ff30ca6a36fada for region: 315e13e868a7f68a3ce37802b2f6312d 2023-04-12 02:54:08,770 DEBUG [StoreFileSplitter-pool-9] assignment.SplitTableRegionProcedure(787): pid=28 splitting complete for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf1/2ca08ea408834a75bc73ef0d3d78fa73 for region: 315e13e868a7f68a3ce37802b2f6312d 2023-04-12 02:54:08,777 DEBUG [StoreFileSplitter-pool-2] assignment.SplitTableRegionProcedure(787): pid=28 splitting complete for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf2/3a80c419c6b94682b44cf75bc8aef7bd for region: 315e13e868a7f68a3ce37802b2f6312d 2023-04-12 02:54:08,808 DEBUG [StoreFileSplitter-pool-12] assignment.SplitTableRegionProcedure(787): pid=28 splitting complete for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf1/713becfd985e40a8892fbc1f6fe6bdb4 for region: 315e13e868a7f68a3ce37802b2f6312d 2023-04-12 02:54:08,824 DEBUG [StoreFileSplitter-pool-10] assignment.SplitTableRegionProcedure(787): pid=28 splitting complete for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf1/2fd181776c3749c18358aebdfc597014 for region: 315e13e868a7f68a3ce37802b2f6312d 2023-04-12 02:54:08,824 DEBUG [StoreFileSplitter-pool-8] assignment.SplitTableRegionProcedure(787): pid=28 splitting complete for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf1/1cabdcde78c14f6a8ec70790c1ae4cc4 for region: 315e13e868a7f68a3ce37802b2f6312d 2023-04-12 02:54:08,824 DEBUG [StoreFileSplitter-pool-4] assignment.SplitTableRegionProcedure(787): pid=28 splitting complete for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf2/d253c9057ec44edbad8b758da6e18d8e for region: 315e13e868a7f68a3ce37802b2f6312d 2023-04-12 02:54:08,829 DEBUG [StoreFileSplitter-pool-7] assignment.SplitTableRegionProcedure(787): pid=28 splitting complete for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf1/0deaf4065ec441859d88faeb52cfa266 for region: 315e13e868a7f68a3ce37802b2f6312d 2023-04-12 02:54:08,841 DEBUG [StoreFileSplitter-pool-5] assignment.SplitTableRegionProcedure(787): pid=28 splitting complete for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf2/d5ab6ea45d1b4298ae7d38df0de5a43f for region: 315e13e868a7f68a3ce37802b2f6312d 2023-04-12 02:54:08,853 DEBUG [StoreFileSplitter-pool-11] assignment.SplitTableRegionProcedure(787): pid=28 splitting complete for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf1/55bdd10dae634a008e897b8e86048f3c for region: 315e13e868a7f68a3ce37802b2f6312d 2023-04-12 02:54:08,857 DEBUG [StoreFileSplitter-pool-13] assignment.SplitTableRegionProcedure(787): pid=28 splitting complete for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf1/7e0f8c9c61e247ecbfc4c27206fae340 for region: 315e13e868a7f68a3ce37802b2f6312d 2023-04-12 02:54:09,209 DEBUG [StoreFileSplitter-pool-6] assignment.SplitTableRegionProcedure(787): pid=28 splitting complete for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf2/f13ea6a8012d44709becae664369d2d0 for region: 315e13e868a7f68a3ce37802b2f6312d 2023-04-12 02:54:09,209 DEBUG [PEWorker-1] assignment.SplitTableRegionProcedure(755): pid=28 split storefiles for region 315e13e868a7f68a3ce37802b2f6312d Daughter A: 14 storefiles, Daughter B: 14 storefiles. 2023-04-12 02:54:09,270 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-04-12 02:54:09,277 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/4e1092f3c00484f8f09c730861225d43/recovered.edits/85.seqid, newMaxSeqId=85, maxSeqId=-1 2023-04-12 02:54:09,281 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/291734c00d212fd203b04cf8872d4649/recovered.edits/85.seqid, newMaxSeqId=85, maxSeqId=-1 2023-04-12 02:54:09,282 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-04-12 02:54:09,282 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_BEFORE_META, locked=true; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=315e13e868a7f68a3ce37802b2f6312d, daughterA=4e1092f3c00484f8f09c730861225d43, daughterB=291734c00d212fd203b04cf8872d4649 2023-04-12 02:54:09,282 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-04-12 02:54:09,282 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_BEFORE_META, locked=true; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=315e13e868a7f68a3ce37802b2f6312d, daughterA=4e1092f3c00484f8f09c730861225d43, daughterB=291734c00d212fd203b04cf8872d4649 at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-04-12 02:54:09,332 INFO [Listener at localhost/46667] procedure.MasterProcedureTestingUtility(417): Restart 6 exec state=pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_BEFORE_META, locked=true; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=315e13e868a7f68a3ce37802b2f6312d, daughterA=4e1092f3c00484f8f09c730861225d43, daughterB=291734c00d212fd203b04cf8872d4649 2023-04-12 02:54:09,332 INFO [Listener at localhost/46667] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-04-12 02:54:09,332 INFO [Listener at localhost/46667] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-04-12 02:54:09,332 INFO [Listener at localhost/46667] assignment.AssignmentManager(315): Stopping assignment manager 2023-04-12 02:54:09,333 ERROR [Listener at localhost/46667] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-04-12 02:54:09,333 INFO [Listener at localhost/46667] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-04-12 02:54:09,333 INFO [Listener at localhost/46667] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-04-12 02:54:09,333 INFO [Listener at localhost/46667] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-04-12 02:54:09,333 INFO [Listener at localhost/46667] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-04-12 02:54:09,334 INFO [Listener at localhost/46667] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-04-12 02:54:09,335 DEBUG [Listener at localhost/46667] store.ProcedureTree(114): Procedure Procedure(pid=28, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[0, 1, 2, 7, 8] 2023-04-12 02:54:09,335 DEBUG [Listener at localhost/46667] store.ProcedureTree(114): Procedure Procedure(pid=29, ppid=28, class=org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure) stack ids=[3, 6] 2023-04-12 02:54:09,335 DEBUG [Listener at localhost/46667] store.ProcedureTree(114): Procedure Procedure(pid=30, ppid=29, class=org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure) stack ids=[4, 5] 2023-04-12 02:54:09,335 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-04-12 02:54:09,336 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-04-12 02:54:09,336 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-04-12 02:54:09,337 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-04-12 02:54:09,337 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-12 02:54:09,338 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=cc9012ec0b656baa24d04368fbe026c4, daughterA=e2a9b8fe15a781798c0b49e48ff5aad7, daughterB=6f55b5d6a460bd8c7569dee22f049104 2023-04-12 02:54:09,338 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-12 02:54:09,338 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-12 02:54:09,338 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-04-12 02:54:09,339 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(468): Loading pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_WRITE_MAX_SEQUENCE_ID_FILE; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=315e13e868a7f68a3ce37802b2f6312d, daughterA=4e1092f3c00484f8f09c730861225d43, daughterB=291734c00d212fd203b04cf8872d4649 2023-04-12 02:54:09,339 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(468): Loading pid=29, ppid=28, state=SUCCESS; TransitRegionStateProcedure table=testSplitWithoutPONR, region=315e13e868a7f68a3ce37802b2f6312d, UNASSIGN 2023-04-12 02:54:09,340 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(468): Loading pid=30, ppid=29, state=SUCCESS; CloseRegionProcedure 315e13e868a7f68a3ce37802b2f6312d, server=jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:09,340 DEBUG [Listener at localhost/46667] procedure2.Procedure(966): pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_WRITE_MAX_SEQUENCE_ID_FILE; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=315e13e868a7f68a3ce37802b2f6312d, daughterA=4e1092f3c00484f8f09c730861225d43, daughterB=291734c00d212fd203b04cf8872d4649 held the lock before restarting, call acquireLock to restore it. 2023-04-12 02:54:09,340 INFO [Listener at localhost/46667] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_WRITE_MAX_SEQUENCE_ID_FILE; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=315e13e868a7f68a3ce37802b2f6312d, daughterA=4e1092f3c00484f8f09c730861225d43, daughterB=291734c00d212fd203b04cf8872d4649 2023-04-12 02:54:09,340 INFO [Listener at localhost/46667] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_WRITE_MAX_SEQUENCE_ID_FILE; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=315e13e868a7f68a3ce37802b2f6312d, daughterA=4e1092f3c00484f8f09c730861225d43, daughterB=291734c00d212fd203b04cf8872d4649 2023-04-12 02:54:09,340 INFO [Listener at localhost/46667] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_WRITE_MAX_SEQUENCE_ID_FILE; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=315e13e868a7f68a3ce37802b2f6312d, daughterA=4e1092f3c00484f8f09c730861225d43, daughterB=291734c00d212fd203b04cf8872d4649 2023-04-12 02:54:09,340 DEBUG [Listener at localhost/46667] procedure2.Procedure(946): pid=29, ppid=28, state=SUCCESS; TransitRegionStateProcedure table=testSplitWithoutPONR, region=315e13e868a7f68a3ce37802b2f6312d, UNASSIGN didn't hold the lock before restarting, skip acquiring lock. 2023-04-12 02:54:09,340 DEBUG [Listener at localhost/46667] procedure2.Procedure(946): pid=30, ppid=29, state=SUCCESS; CloseRegionProcedure 315e13e868a7f68a3ce37802b2f6312d, server=jenkins-hbase4.apache.org,34579,1681268031719 didn't hold the lock before restarting, skip acquiring lock. 2023-04-12 02:54:09,340 INFO [Listener at localhost/46667] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 6 msec 2023-04-12 02:54:09,341 DEBUG [Listener at localhost/46667] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-04-12 02:54:09,342 DEBUG [Listener at localhost/46667] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase4.apache.org,44679,1681268031772, table=hbase:meta, region=1588230740 2023-04-12 02:54:09,359 INFO [Listener at localhost/46667] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1681268079359 2023-04-12 02:54:09,361 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_WRITE_MAX_SEQUENCE_ID_FILE; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=315e13e868a7f68a3ce37802b2f6312d, daughterA=4e1092f3c00484f8f09c730861225d43, daughterB=291734c00d212fd203b04cf8872d4649 2023-04-12 02:54:09,361 DEBUG [Listener at localhost/46667] assignment.AssignmentManager(1517): Joining cluster... 2023-04-12 02:54:09,364 DEBUG [Listener at localhost/46667] assignment.RegionStateStore(147): Load hbase:meta entry region=1f255e980a31bb266f2206a30aa2ae3f, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,44679,1681268031772, regionLocation=jenkins-hbase4.apache.org,44679,1681268031772, openSeqNum=2 2023-04-12 02:54:09,365 DEBUG [Listener at localhost/46667] assignment.RegionStateStore(147): Load hbase:meta entry region=315e13e868a7f68a3ce37802b2f6312d, regionState=CLOSED, lastHost=jenkins-hbase4.apache.org,34579,1681268031719, regionLocation=jenkins-hbase4.apache.org,34579,1681268031719, openSeqNum=2 2023-04-12 02:54:09,365 INFO [Listener at localhost/46667] assignment.AssignmentManager(1529): Number of RegionServers=3 2023-04-12 02:54:09,365 INFO [Listener at localhost/46667] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1681268109365 2023-04-12 02:54:09,365 INFO [Listener at localhost/46667] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1681268169365 2023-04-12 02:54:09,365 INFO [Listener at localhost/46667] assignment.AssignmentManager(1536): Joined the cluster in 3 msec 2023-04-12 02:54:09,368 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-04-12 02:54:09,371 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-04-12 02:54:09,371 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_UPDATE_META, locked=true; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=315e13e868a7f68a3ce37802b2f6312d, daughterA=4e1092f3c00484f8f09c730861225d43, daughterB=291734c00d212fd203b04cf8872d4649 2023-04-12 02:54:09,371 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-04-12 02:54:09,371 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_UPDATE_META, locked=true; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=315e13e868a7f68a3ce37802b2f6312d, daughterA=4e1092f3c00484f8f09c730861225d43, daughterB=291734c00d212fd203b04cf8872d4649 at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-04-12 02:54:09,615 WARN [Listener at localhost/46667] procedure2.ProcedureTestingUtility(193): Set Kill before store update to: false 2023-04-12 02:54:09,615 INFO [Listener at localhost/46667] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-04-12 02:54:09,615 INFO [Listener at localhost/46667] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-04-12 02:54:09,616 INFO [Listener at localhost/46667] assignment.AssignmentManager(315): Stopping assignment manager 2023-04-12 02:54:09,616 ERROR [Listener at localhost/46667] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-04-12 02:54:09,616 INFO [Listener at localhost/46667] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-04-12 02:54:09,616 INFO [Listener at localhost/46667] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-04-12 02:54:09,616 INFO [Listener at localhost/46667] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-04-12 02:54:09,616 INFO [Listener at localhost/46667] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-04-12 02:54:09,617 INFO [Listener at localhost/46667] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-04-12 02:54:09,618 DEBUG [Listener at localhost/46667] store.ProcedureTree(114): Procedure Procedure(pid=28, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[0, 1, 2, 7, 8, 9] 2023-04-12 02:54:09,619 DEBUG [Listener at localhost/46667] store.ProcedureTree(114): Procedure Procedure(pid=29, ppid=28, class=org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure) stack ids=[3, 6] 2023-04-12 02:54:09,619 DEBUG [Listener at localhost/46667] store.ProcedureTree(114): Procedure Procedure(pid=30, ppid=29, class=org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure) stack ids=[4, 5] 2023-04-12 02:54:09,619 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-04-12 02:54:09,620 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-04-12 02:54:09,620 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-04-12 02:54:09,620 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-04-12 02:54:09,620 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-12 02:54:09,621 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=cc9012ec0b656baa24d04368fbe026c4, daughterA=e2a9b8fe15a781798c0b49e48ff5aad7, daughterB=6f55b5d6a460bd8c7569dee22f049104 2023-04-12 02:54:09,621 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-12 02:54:09,621 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-12 02:54:09,622 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-04-12 02:54:09,622 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(468): Loading pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_BEFORE_META; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=315e13e868a7f68a3ce37802b2f6312d, daughterA=4e1092f3c00484f8f09c730861225d43, daughterB=291734c00d212fd203b04cf8872d4649 2023-04-12 02:54:09,623 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(468): Loading pid=29, ppid=28, state=SUCCESS; TransitRegionStateProcedure table=testSplitWithoutPONR, region=315e13e868a7f68a3ce37802b2f6312d, UNASSIGN 2023-04-12 02:54:09,623 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(468): Loading pid=30, ppid=29, state=SUCCESS; CloseRegionProcedure 315e13e868a7f68a3ce37802b2f6312d, server=jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:09,623 DEBUG [Listener at localhost/46667] procedure2.Procedure(966): pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_BEFORE_META; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=315e13e868a7f68a3ce37802b2f6312d, daughterA=4e1092f3c00484f8f09c730861225d43, daughterB=291734c00d212fd203b04cf8872d4649 held the lock before restarting, call acquireLock to restore it. 2023-04-12 02:54:09,623 INFO [Listener at localhost/46667] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_BEFORE_META; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=315e13e868a7f68a3ce37802b2f6312d, daughterA=4e1092f3c00484f8f09c730861225d43, daughterB=291734c00d212fd203b04cf8872d4649 2023-04-12 02:54:09,623 INFO [Listener at localhost/46667] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_BEFORE_META; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=315e13e868a7f68a3ce37802b2f6312d, daughterA=4e1092f3c00484f8f09c730861225d43, daughterB=291734c00d212fd203b04cf8872d4649 2023-04-12 02:54:09,623 INFO [Listener at localhost/46667] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_BEFORE_META; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=315e13e868a7f68a3ce37802b2f6312d, daughterA=4e1092f3c00484f8f09c730861225d43, daughterB=291734c00d212fd203b04cf8872d4649 2023-04-12 02:54:09,623 DEBUG [Listener at localhost/46667] procedure2.Procedure(946): pid=29, ppid=28, state=SUCCESS; TransitRegionStateProcedure table=testSplitWithoutPONR, region=315e13e868a7f68a3ce37802b2f6312d, UNASSIGN didn't hold the lock before restarting, skip acquiring lock. 2023-04-12 02:54:09,623 DEBUG [Listener at localhost/46667] procedure2.Procedure(946): pid=30, ppid=29, state=SUCCESS; CloseRegionProcedure 315e13e868a7f68a3ce37802b2f6312d, server=jenkins-hbase4.apache.org,34579,1681268031719 didn't hold the lock before restarting, skip acquiring lock. 2023-04-12 02:54:09,624 INFO [Listener at localhost/46667] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 6 msec 2023-04-12 02:54:09,624 DEBUG [Listener at localhost/46667] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-04-12 02:54:09,625 DEBUG [Listener at localhost/46667] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase4.apache.org,44679,1681268031772, table=hbase:meta, region=1588230740 2023-04-12 02:54:09,625 INFO [Listener at localhost/46667] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1681268079625 2023-04-12 02:54:09,625 DEBUG [Listener at localhost/46667] assignment.AssignmentManager(1517): Joining cluster... 2023-04-12 02:54:09,627 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_BEFORE_META; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=315e13e868a7f68a3ce37802b2f6312d, daughterA=4e1092f3c00484f8f09c730861225d43, daughterB=291734c00d212fd203b04cf8872d4649 2023-04-12 02:54:09,630 DEBUG [Listener at localhost/46667] assignment.RegionStateStore(147): Load hbase:meta entry region=1f255e980a31bb266f2206a30aa2ae3f, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,44679,1681268031772, regionLocation=jenkins-hbase4.apache.org,44679,1681268031772, openSeqNum=2 2023-04-12 02:54:09,630 DEBUG [Listener at localhost/46667] assignment.RegionStateStore(147): Load hbase:meta entry region=315e13e868a7f68a3ce37802b2f6312d, regionState=CLOSED, lastHost=jenkins-hbase4.apache.org,34579,1681268031719, regionLocation=jenkins-hbase4.apache.org,34579,1681268031719, openSeqNum=2 2023-04-12 02:54:09,630 INFO [Listener at localhost/46667] assignment.AssignmentManager(1529): Number of RegionServers=3 2023-04-12 02:54:09,630 INFO [Listener at localhost/46667] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1681268109630 2023-04-12 02:54:09,630 INFO [Listener at localhost/46667] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1681268169630 2023-04-12 02:54:09,630 INFO [Listener at localhost/46667] assignment.AssignmentManager(1536): Joined the cluster in 4 msec 2023-04-12 02:54:09,635 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":3,"row":"testSplitWithoutPONR,,1681268041279.315e13e868a7f68a3ce37802b2f6312d.","families":{"info":[{"qualifier":"regioninfo","vlen":54,"tag":[],"timestamp":"1681268049634"},{"qualifier":"splitA","vlen":56,"tag":[],"timestamp":"1681268049634"},{"qualifier":"splitB","vlen":56,"tag":[],"timestamp":"1681268049634"}]},"ts":"1681268049634"} 2023-04-12 02:54:09,635 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":3,"row":"testSplitWithoutPONR,,1681268044634.4e1092f3c00484f8f09c730861225d43.","families":{"info":[{"qualifier":"regioninfo","vlen":56,"tag":[],"timestamp":"1681268049634"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681268049634"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681268049634"}]},"ts":"1681268049634"} 2023-04-12 02:54:09,635 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":3,"row":"testSplitWithoutPONR,41,1681268044634.291734c00d212fd203b04cf8872d4649.","families":{"info":[{"qualifier":"regioninfo","vlen":56,"tag":[],"timestamp":"1681268049634"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681268049634"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681268049634"}]},"ts":"1681268049634"} 2023-04-12 02:54:09,642 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=31, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSplitWithoutPONR, region=4e1092f3c00484f8f09c730861225d43, ASSIGN}, {pid=32, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSplitWithoutPONR, region=291734c00d212fd203b04cf8872d4649, ASSIGN}] 2023-04-12 02:54:09,645 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=32, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSplitWithoutPONR, region=291734c00d212fd203b04cf8872d4649, ASSIGN 2023-04-12 02:54:09,646 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=32, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testSplitWithoutPONR, region=291734c00d212fd203b04cf8872d4649, ASSIGN; state=SPLITTING_NEW, location=jenkins-hbase4.apache.org,34579,1681268031719; forceNewPlan=false, retain=false 2023-04-12 02:54:09,648 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=31, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSplitWithoutPONR, region=4e1092f3c00484f8f09c730861225d43, ASSIGN 2023-04-12 02:54:09,649 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=31, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testSplitWithoutPONR, region=4e1092f3c00484f8f09c730861225d43, ASSIGN; state=SPLITTING_NEW, location=jenkins-hbase4.apache.org,34579,1681268031719; forceNewPlan=false, retain=false 2023-04-12 02:54:09,796 INFO [jenkins-hbase4:40281] balancer.BaseLoadBalancer(1545): Reassigned 2 regions. 2 retained the pre-restart assignment. 2023-04-12 02:54:09,798 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=31 updating hbase:meta row=4e1092f3c00484f8f09c730861225d43, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:09,798 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitWithoutPONR,,1681268044634.4e1092f3c00484f8f09c730861225d43.","families":{"info":[{"qualifier":"regioninfo","vlen":56,"tag":[],"timestamp":"1681268049798"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681268049798"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681268049798"}]},"ts":"1681268049798"} 2023-04-12 02:54:09,800 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=33, ppid=31, state=RUNNABLE; OpenRegionProcedure 4e1092f3c00484f8f09c730861225d43, server=jenkins-hbase4.apache.org,34579,1681268031719}] 2023-04-12 02:54:09,807 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=32 updating hbase:meta row=291734c00d212fd203b04cf8872d4649, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:09,807 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitWithoutPONR,41,1681268044634.291734c00d212fd203b04cf8872d4649.","families":{"info":[{"qualifier":"regioninfo","vlen":56,"tag":[],"timestamp":"1681268049807"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681268049807"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681268049807"}]},"ts":"1681268049807"} 2023-04-12 02:54:09,813 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=34, ppid=32, state=RUNNABLE; OpenRegionProcedure 291734c00d212fd203b04cf8872d4649, server=jenkins-hbase4.apache.org,34579,1681268031719}] 2023-04-12 02:54:09,959 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testSplitWithoutPONR,,1681268044634.4e1092f3c00484f8f09c730861225d43. 2023-04-12 02:54:09,959 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 4e1092f3c00484f8f09c730861225d43, NAME => 'testSplitWithoutPONR,,1681268044634.4e1092f3c00484f8f09c730861225d43.', STARTKEY => '', ENDKEY => '41'} 2023-04-12 02:54:09,960 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(174): System coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro loaded, priority=536870911. 2023-04-12 02:54:09,960 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testSplitWithoutPONR 4e1092f3c00484f8f09c730861225d43 2023-04-12 02:54:09,960 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated testSplitWithoutPONR,,1681268044634.4e1092f3c00484f8f09c730861225d43.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-12 02:54:09,960 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 4e1092f3c00484f8f09c730861225d43 2023-04-12 02:54:09,960 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 4e1092f3c00484f8f09c730861225d43 2023-04-12 02:54:09,962 INFO [StoreOpener-4e1092f3c00484f8f09c730861225d43-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf1 of region 4e1092f3c00484f8f09c730861225d43 2023-04-12 02:54:09,963 DEBUG [StoreOpener-4e1092f3c00484f8f09c730861225d43-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/4e1092f3c00484f8f09c730861225d43/cf1 2023-04-12 02:54:09,963 DEBUG [StoreOpener-4e1092f3c00484f8f09c730861225d43-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/4e1092f3c00484f8f09c730861225d43/cf1 2023-04-12 02:54:09,964 INFO [StoreOpener-4e1092f3c00484f8f09c730861225d43-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 4e1092f3c00484f8f09c730861225d43 columnFamilyName cf1 2023-04-12 02:54:09,986 DEBUG [StoreOpener-4e1092f3c00484f8f09c730861225d43-1] regionserver.HStore(539): loaded hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/4e1092f3c00484f8f09c730861225d43/cf1/0deaf4065ec441859d88faeb52cfa266.315e13e868a7f68a3ce37802b2f6312d->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf1/0deaf4065ec441859d88faeb52cfa266-bottom 2023-04-12 02:54:09,991 DEBUG [StoreOpener-4e1092f3c00484f8f09c730861225d43-1] regionserver.HStore(539): loaded hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/4e1092f3c00484f8f09c730861225d43/cf1/1cabdcde78c14f6a8ec70790c1ae4cc4.315e13e868a7f68a3ce37802b2f6312d->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf1/1cabdcde78c14f6a8ec70790c1ae4cc4-bottom 2023-04-12 02:54:10,002 DEBUG [StoreOpener-4e1092f3c00484f8f09c730861225d43-1] regionserver.HStore(539): loaded hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/4e1092f3c00484f8f09c730861225d43/cf1/2ca08ea408834a75bc73ef0d3d78fa73.315e13e868a7f68a3ce37802b2f6312d->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf1/2ca08ea408834a75bc73ef0d3d78fa73-bottom 2023-04-12 02:54:10,012 DEBUG [StoreOpener-4e1092f3c00484f8f09c730861225d43-1] regionserver.HStore(539): loaded hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/4e1092f3c00484f8f09c730861225d43/cf1/2fd181776c3749c18358aebdfc597014.315e13e868a7f68a3ce37802b2f6312d->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf1/2fd181776c3749c18358aebdfc597014-bottom 2023-04-12 02:54:10,015 DEBUG [StoreOpener-4e1092f3c00484f8f09c730861225d43-1] regionserver.HStore(539): loaded hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/4e1092f3c00484f8f09c730861225d43/cf1/55bdd10dae634a008e897b8e86048f3c.315e13e868a7f68a3ce37802b2f6312d->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf1/55bdd10dae634a008e897b8e86048f3c-bottom 2023-04-12 02:54:10,021 DEBUG [StoreOpener-4e1092f3c00484f8f09c730861225d43-1] regionserver.HStore(539): loaded hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/4e1092f3c00484f8f09c730861225d43/cf1/713becfd985e40a8892fbc1f6fe6bdb4.315e13e868a7f68a3ce37802b2f6312d->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf1/713becfd985e40a8892fbc1f6fe6bdb4-bottom 2023-04-12 02:54:10,028 DEBUG [StoreOpener-4e1092f3c00484f8f09c730861225d43-1] regionserver.HStore(539): loaded hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/4e1092f3c00484f8f09c730861225d43/cf1/7e0f8c9c61e247ecbfc4c27206fae340.315e13e868a7f68a3ce37802b2f6312d->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf1/7e0f8c9c61e247ecbfc4c27206fae340-bottom 2023-04-12 02:54:10,029 INFO [StoreOpener-4e1092f3c00484f8f09c730861225d43-1] regionserver.HStore(310): Store=4e1092f3c00484f8f09c730861225d43/cf1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-12 02:54:10,029 INFO [StoreOpener-4e1092f3c00484f8f09c730861225d43-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf2 of region 4e1092f3c00484f8f09c730861225d43 2023-04-12 02:54:10,030 DEBUG [StoreOpener-4e1092f3c00484f8f09c730861225d43-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/4e1092f3c00484f8f09c730861225d43/cf2 2023-04-12 02:54:10,030 DEBUG [StoreOpener-4e1092f3c00484f8f09c730861225d43-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/4e1092f3c00484f8f09c730861225d43/cf2 2023-04-12 02:54:10,031 INFO [StoreOpener-4e1092f3c00484f8f09c730861225d43-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 4e1092f3c00484f8f09c730861225d43 columnFamilyName cf2 2023-04-12 02:54:10,050 DEBUG [StoreOpener-4e1092f3c00484f8f09c730861225d43-1] regionserver.HStore(539): loaded hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/4e1092f3c00484f8f09c730861225d43/cf2/0b28a4ef23e442f38cfdf4760e6f9480.315e13e868a7f68a3ce37802b2f6312d->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf2/0b28a4ef23e442f38cfdf4760e6f9480-bottom 2023-04-12 02:54:10,055 DEBUG [StoreOpener-4e1092f3c00484f8f09c730861225d43-1] regionserver.HStore(539): loaded hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/4e1092f3c00484f8f09c730861225d43/cf2/2540a68ffbdf4440b5ff30ca6a36fada.315e13e868a7f68a3ce37802b2f6312d->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf2/2540a68ffbdf4440b5ff30ca6a36fada-bottom 2023-04-12 02:54:10,061 DEBUG [StoreOpener-4e1092f3c00484f8f09c730861225d43-1] regionserver.HStore(539): loaded hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/4e1092f3c00484f8f09c730861225d43/cf2/3a80c419c6b94682b44cf75bc8aef7bd.315e13e868a7f68a3ce37802b2f6312d->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf2/3a80c419c6b94682b44cf75bc8aef7bd-bottom 2023-04-12 02:54:10,066 DEBUG [StoreOpener-4e1092f3c00484f8f09c730861225d43-1] regionserver.HStore(539): loaded hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/4e1092f3c00484f8f09c730861225d43/cf2/babf83b74cec452290ff19cd45eb4713.315e13e868a7f68a3ce37802b2f6312d->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf2/babf83b74cec452290ff19cd45eb4713-bottom 2023-04-12 02:54:10,076 DEBUG [StoreOpener-4e1092f3c00484f8f09c730861225d43-1] regionserver.HStore(539): loaded hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/4e1092f3c00484f8f09c730861225d43/cf2/d253c9057ec44edbad8b758da6e18d8e.315e13e868a7f68a3ce37802b2f6312d->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf2/d253c9057ec44edbad8b758da6e18d8e-bottom 2023-04-12 02:54:10,080 DEBUG [StoreOpener-4e1092f3c00484f8f09c730861225d43-1] regionserver.HStore(539): loaded hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/4e1092f3c00484f8f09c730861225d43/cf2/d5ab6ea45d1b4298ae7d38df0de5a43f.315e13e868a7f68a3ce37802b2f6312d->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf2/d5ab6ea45d1b4298ae7d38df0de5a43f-bottom 2023-04-12 02:54:10,089 DEBUG [StoreOpener-4e1092f3c00484f8f09c730861225d43-1] regionserver.HStore(539): loaded hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/4e1092f3c00484f8f09c730861225d43/cf2/f13ea6a8012d44709becae664369d2d0.315e13e868a7f68a3ce37802b2f6312d->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf2/f13ea6a8012d44709becae664369d2d0-bottom 2023-04-12 02:54:10,089 INFO [StoreOpener-4e1092f3c00484f8f09c730861225d43-1] regionserver.HStore(310): Store=4e1092f3c00484f8f09c730861225d43/cf2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-12 02:54:10,090 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/4e1092f3c00484f8f09c730861225d43 2023-04-12 02:54:10,091 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/4e1092f3c00484f8f09c730861225d43 2023-04-12 02:54:10,094 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testSplitWithoutPONR descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-04-12 02:54:10,095 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 4e1092f3c00484f8f09c730861225d43 2023-04-12 02:54:10,096 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 4e1092f3c00484f8f09c730861225d43; next sequenceid=86; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=62638592, jitterRate=-0.06661224365234375}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-04-12 02:54:10,096 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 4e1092f3c00484f8f09c730861225d43: 2023-04-12 02:54:10,097 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testSplitWithoutPONR,,1681268044634.4e1092f3c00484f8f09c730861225d43., pid=33, masterSystemTime=1681268049955 2023-04-12 02:54:10,098 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: Opening Region; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 02:54:10,099 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: Opening Region; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 02:54:10,100 DEBUG [RS:0;jenkins-hbase4:34579-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-12 02:54:10,100 DEBUG [RS:0;jenkins-hbase4:34579-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-12 02:54:10,100 INFO [RS:0;jenkins-hbase4:34579-longCompactions-0] regionserver.HStore(1898): Keeping/Overriding Compaction request priority to -2147482648 for CF cf2 since it belongs to recently split daughter region testSplitWithoutPONR,,1681268044634.4e1092f3c00484f8f09c730861225d43. 2023-04-12 02:54:10,100 DEBUG [RS:0;jenkins-hbase4:34579-longCompactions-0] regionserver.HStore(1912): 4e1092f3c00484f8f09c730861225d43/cf2 is initiating minor compaction (all files) 2023-04-12 02:54:10,100 INFO [RS:0;jenkins-hbase4:34579-shortCompactions-0] regionserver.HStore(1898): Keeping/Overriding Compaction request priority to -2147482648 for CF cf1 since it belongs to recently split daughter region testSplitWithoutPONR,,1681268044634.4e1092f3c00484f8f09c730861225d43. 2023-04-12 02:54:10,100 DEBUG [RS:0;jenkins-hbase4:34579-shortCompactions-0] regionserver.HStore(1912): 4e1092f3c00484f8f09c730861225d43/cf1 is initiating minor compaction (all files) 2023-04-12 02:54:10,100 INFO [RS:0;jenkins-hbase4:34579-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 4e1092f3c00484f8f09c730861225d43/cf2 in testSplitWithoutPONR,,1681268044634.4e1092f3c00484f8f09c730861225d43. 2023-04-12 02:54:10,100 INFO [RS:0;jenkins-hbase4:34579-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 4e1092f3c00484f8f09c730861225d43/cf1 in testSplitWithoutPONR,,1681268044634.4e1092f3c00484f8f09c730861225d43. 2023-04-12 02:54:10,101 INFO [RS:0;jenkins-hbase4:34579-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/4e1092f3c00484f8f09c730861225d43/cf2/0b28a4ef23e442f38cfdf4760e6f9480.315e13e868a7f68a3ce37802b2f6312d->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf2/0b28a4ef23e442f38cfdf4760e6f9480-bottom, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/4e1092f3c00484f8f09c730861225d43/cf2/2540a68ffbdf4440b5ff30ca6a36fada.315e13e868a7f68a3ce37802b2f6312d->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf2/2540a68ffbdf4440b5ff30ca6a36fada-bottom, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/4e1092f3c00484f8f09c730861225d43/cf2/babf83b74cec452290ff19cd45eb4713.315e13e868a7f68a3ce37802b2f6312d->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf2/babf83b74cec452290ff19cd45eb4713-bottom, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/4e1092f3c00484f8f09c730861225d43/cf2/3a80c419c6b94682b44cf75bc8aef7bd.315e13e868a7f68a3ce37802b2f6312d->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf2/3a80c419c6b94682b44cf75bc8aef7bd-bottom, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/4e1092f3c00484f8f09c730861225d43/cf2/f13ea6a8012d44709becae664369d2d0.315e13e868a7f68a3ce37802b2f6312d->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf2/f13ea6a8012d44709becae664369d2d0-bottom, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/4e1092f3c00484f8f09c730861225d43/cf2/d5ab6ea45d1b4298ae7d38df0de5a43f.315e13e868a7f68a3ce37802b2f6312d->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf2/d5ab6ea45d1b4298ae7d38df0de5a43f-bottom, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/4e1092f3c00484f8f09c730861225d43/cf2/d253c9057ec44edbad8b758da6e18d8e.315e13e868a7f68a3ce37802b2f6312d->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf2/d253c9057ec44edbad8b758da6e18d8e-bottom] into tmpdir=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/4e1092f3c00484f8f09c730861225d43/.tmp, totalSize=34.9 K 2023-04-12 02:54:10,101 INFO [RS:0;jenkins-hbase4:34579-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/4e1092f3c00484f8f09c730861225d43/cf1/1cabdcde78c14f6a8ec70790c1ae4cc4.315e13e868a7f68a3ce37802b2f6312d->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf1/1cabdcde78c14f6a8ec70790c1ae4cc4-bottom, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/4e1092f3c00484f8f09c730861225d43/cf1/2fd181776c3749c18358aebdfc597014.315e13e868a7f68a3ce37802b2f6312d->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf1/2fd181776c3749c18358aebdfc597014-bottom, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/4e1092f3c00484f8f09c730861225d43/cf1/55bdd10dae634a008e897b8e86048f3c.315e13e868a7f68a3ce37802b2f6312d->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf1/55bdd10dae634a008e897b8e86048f3c-bottom, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/4e1092f3c00484f8f09c730861225d43/cf1/7e0f8c9c61e247ecbfc4c27206fae340.315e13e868a7f68a3ce37802b2f6312d->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf1/7e0f8c9c61e247ecbfc4c27206fae340-bottom, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/4e1092f3c00484f8f09c730861225d43/cf1/2ca08ea408834a75bc73ef0d3d78fa73.315e13e868a7f68a3ce37802b2f6312d->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf1/2ca08ea408834a75bc73ef0d3d78fa73-bottom, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/4e1092f3c00484f8f09c730861225d43/cf1/713becfd985e40a8892fbc1f6fe6bdb4.315e13e868a7f68a3ce37802b2f6312d->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf1/713becfd985e40a8892fbc1f6fe6bdb4-bottom, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/4e1092f3c00484f8f09c730861225d43/cf1/0deaf4065ec441859d88faeb52cfa266.315e13e868a7f68a3ce37802b2f6312d->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf1/0deaf4065ec441859d88faeb52cfa266-bottom] into tmpdir=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/4e1092f3c00484f8f09c730861225d43/.tmp, totalSize=34.9 K 2023-04-12 02:54:10,101 DEBUG [RS:0;jenkins-hbase4:34579-longCompactions-0] compactions.Compactor(207): Compacting 0b28a4ef23e442f38cfdf4760e6f9480.315e13e868a7f68a3ce37802b2f6312d, keycount=1, bloomtype=ROW, size=4.8 K, encoding=NONE, compression=NONE, seqNum=6, earliestPutTs=1681268042050 2023-04-12 02:54:10,102 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testSplitWithoutPONR,,1681268044634.4e1092f3c00484f8f09c730861225d43. 2023-04-12 02:54:10,102 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testSplitWithoutPONR,,1681268044634.4e1092f3c00484f8f09c730861225d43. 2023-04-12 02:54:10,102 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testSplitWithoutPONR,41,1681268044634.291734c00d212fd203b04cf8872d4649. 2023-04-12 02:54:10,102 DEBUG [RS:0;jenkins-hbase4:34579-shortCompactions-0] compactions.Compactor(207): Compacting 1cabdcde78c14f6a8ec70790c1ae4cc4.315e13e868a7f68a3ce37802b2f6312d, keycount=1, bloomtype=ROW, size=4.8 K, encoding=NONE, compression=NONE, seqNum=6, earliestPutTs=1681268042050 2023-04-12 02:54:10,102 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 291734c00d212fd203b04cf8872d4649, NAME => 'testSplitWithoutPONR,41,1681268044634.291734c00d212fd203b04cf8872d4649.', STARTKEY => '41', ENDKEY => ''} 2023-04-12 02:54:10,102 DEBUG [RS:0;jenkins-hbase4:34579-longCompactions-0] compactions.Compactor(207): Compacting 2540a68ffbdf4440b5ff30ca6a36fada.315e13e868a7f68a3ce37802b2f6312d, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=19, earliestPutTs=1681268042410 2023-04-12 02:54:10,103 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(174): System coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro loaded, priority=536870911. 2023-04-12 02:54:10,103 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testSplitWithoutPONR 291734c00d212fd203b04cf8872d4649 2023-04-12 02:54:10,103 DEBUG [RS:0;jenkins-hbase4:34579-shortCompactions-0] compactions.Compactor(207): Compacting 2fd181776c3749c18358aebdfc597014.315e13e868a7f68a3ce37802b2f6312d, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=19, earliestPutTs=1681268042410 2023-04-12 02:54:10,104 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=31 updating hbase:meta row=4e1092f3c00484f8f09c730861225d43, regionState=OPEN, openSeqNum=86, regionLocation=jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:10,104 DEBUG [RS:0;jenkins-hbase4:34579-longCompactions-0] compactions.Compactor(207): Compacting babf83b74cec452290ff19cd45eb4713.315e13e868a7f68a3ce37802b2f6312d, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=32, earliestPutTs=1681268042730 2023-04-12 02:54:10,103 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated testSplitWithoutPONR,41,1681268044634.291734c00d212fd203b04cf8872d4649.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-12 02:54:10,104 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testSplitWithoutPONR,,1681268044634.4e1092f3c00484f8f09c730861225d43.","families":{"info":[{"qualifier":"regioninfo","vlen":56,"tag":[],"timestamp":"1681268050104"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1681268050104"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681268050104"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681268050104"}]},"ts":"1681268050104"} 2023-04-12 02:54:10,104 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 291734c00d212fd203b04cf8872d4649 2023-04-12 02:54:10,104 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 291734c00d212fd203b04cf8872d4649 2023-04-12 02:54:10,105 DEBUG [RS:0;jenkins-hbase4:34579-shortCompactions-0] compactions.Compactor(207): Compacting 55bdd10dae634a008e897b8e86048f3c.315e13e868a7f68a3ce37802b2f6312d, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=32, earliestPutTs=1681268042730 2023-04-12 02:54:10,105 DEBUG [RS:0;jenkins-hbase4:34579-longCompactions-0] compactions.Compactor(207): Compacting 3a80c419c6b94682b44cf75bc8aef7bd.315e13e868a7f68a3ce37802b2f6312d, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=45, earliestPutTs=1681268043069 2023-04-12 02:54:10,106 INFO [StoreOpener-291734c00d212fd203b04cf8872d4649-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf1 of region 291734c00d212fd203b04cf8872d4649 2023-04-12 02:54:10,106 DEBUG [RS:0;jenkins-hbase4:34579-longCompactions-0] compactions.Compactor(207): Compacting f13ea6a8012d44709becae664369d2d0.315e13e868a7f68a3ce37802b2f6312d, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=58, earliestPutTs=1681268043360 2023-04-12 02:54:10,107 DEBUG [RS:0;jenkins-hbase4:34579-shortCompactions-0] compactions.Compactor(207): Compacting 7e0f8c9c61e247ecbfc4c27206fae340.315e13e868a7f68a3ce37802b2f6312d, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=45, earliestPutTs=1681268043069 2023-04-12 02:54:10,107 DEBUG [RS:0;jenkins-hbase4:34579-longCompactions-0] compactions.Compactor(207): Compacting d5ab6ea45d1b4298ae7d38df0de5a43f.315e13e868a7f68a3ce37802b2f6312d, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=71, earliestPutTs=1681268043630 2023-04-12 02:54:10,108 DEBUG [RS:0;jenkins-hbase4:34579-shortCompactions-0] compactions.Compactor(207): Compacting 2ca08ea408834a75bc73ef0d3d78fa73.315e13e868a7f68a3ce37802b2f6312d, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=58, earliestPutTs=1681268043360 2023-04-12 02:54:10,108 DEBUG [StoreOpener-291734c00d212fd203b04cf8872d4649-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/291734c00d212fd203b04cf8872d4649/cf1 2023-04-12 02:54:10,108 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=33, resume processing ppid=31 2023-04-12 02:54:10,109 DEBUG [StoreOpener-291734c00d212fd203b04cf8872d4649-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/291734c00d212fd203b04cf8872d4649/cf1 2023-04-12 02:54:10,109 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=33, ppid=31, state=SUCCESS; OpenRegionProcedure 4e1092f3c00484f8f09c730861225d43, server=jenkins-hbase4.apache.org,34579,1681268031719 in 306 msec 2023-04-12 02:54:10,109 DEBUG [RS:0;jenkins-hbase4:34579-longCompactions-0] compactions.Compactor(207): Compacting d253c9057ec44edbad8b758da6e18d8e.315e13e868a7f68a3ce37802b2f6312d, keycount=4, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=82, earliestPutTs=1681268044369 2023-04-12 02:54:10,109 INFO [StoreOpener-291734c00d212fd203b04cf8872d4649-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 291734c00d212fd203b04cf8872d4649 columnFamilyName cf1 2023-04-12 02:54:10,109 DEBUG [RS:0;jenkins-hbase4:34579-shortCompactions-0] compactions.Compactor(207): Compacting 713becfd985e40a8892fbc1f6fe6bdb4.315e13e868a7f68a3ce37802b2f6312d, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=71, earliestPutTs=1681268043630 2023-04-12 02:54:10,111 DEBUG [RS:0;jenkins-hbase4:34579-shortCompactions-0] compactions.Compactor(207): Compacting 0deaf4065ec441859d88faeb52cfa266.315e13e868a7f68a3ce37802b2f6312d, keycount=4, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=82, earliestPutTs=1681268044369 2023-04-12 02:54:10,112 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=31, ppid=28, state=SUCCESS; TransitRegionStateProcedure table=testSplitWithoutPONR, region=4e1092f3c00484f8f09c730861225d43, ASSIGN in 469 msec 2023-04-12 02:54:10,133 INFO [RS:0;jenkins-hbase4:34579-shortCompactions-0] throttle.PressureAwareThroughputController(145): 4e1092f3c00484f8f09c730861225d43#cf1#compaction#32 average throughput is 0.86 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 02:54:10,136 INFO [RS:0;jenkins-hbase4:34579-longCompactions-0] throttle.PressureAwareThroughputController(145): 4e1092f3c00484f8f09c730861225d43#cf2#compaction#33 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 02:54:10,146 DEBUG [StoreOpener-291734c00d212fd203b04cf8872d4649-1] regionserver.HStore(539): loaded hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/291734c00d212fd203b04cf8872d4649/cf1/0deaf4065ec441859d88faeb52cfa266.315e13e868a7f68a3ce37802b2f6312d->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf1/0deaf4065ec441859d88faeb52cfa266-top 2023-04-12 02:54:10,156 DEBUG [StoreOpener-291734c00d212fd203b04cf8872d4649-1] regionserver.HStore(539): loaded hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/291734c00d212fd203b04cf8872d4649/cf1/1cabdcde78c14f6a8ec70790c1ae4cc4.315e13e868a7f68a3ce37802b2f6312d->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf1/1cabdcde78c14f6a8ec70790c1ae4cc4-top 2023-04-12 02:54:10,161 DEBUG [StoreOpener-291734c00d212fd203b04cf8872d4649-1] regionserver.HStore(539): loaded hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/291734c00d212fd203b04cf8872d4649/cf1/2ca08ea408834a75bc73ef0d3d78fa73.315e13e868a7f68a3ce37802b2f6312d->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf1/2ca08ea408834a75bc73ef0d3d78fa73-top 2023-04-12 02:54:10,217 DEBUG [StoreOpener-291734c00d212fd203b04cf8872d4649-1] regionserver.HStore(539): loaded hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/291734c00d212fd203b04cf8872d4649/cf1/2fd181776c3749c18358aebdfc597014.315e13e868a7f68a3ce37802b2f6312d->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf1/2fd181776c3749c18358aebdfc597014-top 2023-04-12 02:54:10,233 DEBUG [StoreOpener-291734c00d212fd203b04cf8872d4649-1] regionserver.HStore(539): loaded hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/291734c00d212fd203b04cf8872d4649/cf1/55bdd10dae634a008e897b8e86048f3c.315e13e868a7f68a3ce37802b2f6312d->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf1/55bdd10dae634a008e897b8e86048f3c-top 2023-04-12 02:54:10,240 DEBUG [RS:0;jenkins-hbase4:34579-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/4e1092f3c00484f8f09c730861225d43/.tmp/cf1/fcf8238b62664b00a68f527d6561a646 as hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/4e1092f3c00484f8f09c730861225d43/cf1/fcf8238b62664b00a68f527d6561a646 2023-04-12 02:54:10,242 DEBUG [StoreOpener-291734c00d212fd203b04cf8872d4649-1] regionserver.HStore(539): loaded hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/291734c00d212fd203b04cf8872d4649/cf1/713becfd985e40a8892fbc1f6fe6bdb4.315e13e868a7f68a3ce37802b2f6312d->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf1/713becfd985e40a8892fbc1f6fe6bdb4-top 2023-04-12 02:54:10,249 DEBUG [RS:0;jenkins-hbase4:34579-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/4e1092f3c00484f8f09c730861225d43/.tmp/cf2/63fd42795d264c699b7793e9ff2e314f as hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/4e1092f3c00484f8f09c730861225d43/cf2/63fd42795d264c699b7793e9ff2e314f 2023-04-12 02:54:10,255 INFO [RS:0;jenkins-hbase4:34579-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 4e1092f3c00484f8f09c730861225d43/cf2 of 4e1092f3c00484f8f09c730861225d43 into 63fd42795d264c699b7793e9ff2e314f(size=6.1 K), total size for store is 6.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 02:54:10,255 DEBUG [RS:0;jenkins-hbase4:34579-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4e1092f3c00484f8f09c730861225d43: 2023-04-12 02:54:10,255 INFO [RS:0;jenkins-hbase4:34579-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=testSplitWithoutPONR,,1681268044634.4e1092f3c00484f8f09c730861225d43., storeName=4e1092f3c00484f8f09c730861225d43/cf2, priority=9, startTime=1681268050099; duration=0sec 2023-04-12 02:54:10,256 DEBUG [RS:0;jenkins-hbase4:34579-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 02:54:10,260 DEBUG [StoreOpener-291734c00d212fd203b04cf8872d4649-1] regionserver.HStore(539): loaded hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/291734c00d212fd203b04cf8872d4649/cf1/7e0f8c9c61e247ecbfc4c27206fae340.315e13e868a7f68a3ce37802b2f6312d->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf1/7e0f8c9c61e247ecbfc4c27206fae340-top 2023-04-12 02:54:10,260 INFO [StoreOpener-291734c00d212fd203b04cf8872d4649-1] regionserver.HStore(310): Store=291734c00d212fd203b04cf8872d4649/cf1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-12 02:54:10,260 INFO [StoreOpener-291734c00d212fd203b04cf8872d4649-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf2 of region 291734c00d212fd203b04cf8872d4649 2023-04-12 02:54:10,264 DEBUG [StoreOpener-291734c00d212fd203b04cf8872d4649-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/291734c00d212fd203b04cf8872d4649/cf2 2023-04-12 02:54:10,265 DEBUG [StoreOpener-291734c00d212fd203b04cf8872d4649-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/291734c00d212fd203b04cf8872d4649/cf2 2023-04-12 02:54:10,265 INFO [RS:0;jenkins-hbase4:34579-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 4e1092f3c00484f8f09c730861225d43/cf1 of 4e1092f3c00484f8f09c730861225d43 into fcf8238b62664b00a68f527d6561a646(size=6.1 K), total size for store is 6.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 02:54:10,265 DEBUG [RS:0;jenkins-hbase4:34579-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 4e1092f3c00484f8f09c730861225d43: 2023-04-12 02:54:10,265 INFO [RS:0;jenkins-hbase4:34579-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=testSplitWithoutPONR,,1681268044634.4e1092f3c00484f8f09c730861225d43., storeName=4e1092f3c00484f8f09c730861225d43/cf1, priority=9, startTime=1681268050097; duration=0sec 2023-04-12 02:54:10,265 INFO [StoreOpener-291734c00d212fd203b04cf8872d4649-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 291734c00d212fd203b04cf8872d4649 columnFamilyName cf2 2023-04-12 02:54:10,265 DEBUG [RS:0;jenkins-hbase4:34579-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 02:54:10,285 DEBUG [StoreOpener-291734c00d212fd203b04cf8872d4649-1] regionserver.HStore(539): loaded hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/291734c00d212fd203b04cf8872d4649/cf2/0b28a4ef23e442f38cfdf4760e6f9480.315e13e868a7f68a3ce37802b2f6312d->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf2/0b28a4ef23e442f38cfdf4760e6f9480-top 2023-04-12 02:54:10,288 DEBUG [StoreOpener-291734c00d212fd203b04cf8872d4649-1] regionserver.HStore(539): loaded hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/291734c00d212fd203b04cf8872d4649/cf2/2540a68ffbdf4440b5ff30ca6a36fada.315e13e868a7f68a3ce37802b2f6312d->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf2/2540a68ffbdf4440b5ff30ca6a36fada-top 2023-04-12 02:54:10,293 DEBUG [StoreOpener-291734c00d212fd203b04cf8872d4649-1] regionserver.HStore(539): loaded hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/291734c00d212fd203b04cf8872d4649/cf2/3a80c419c6b94682b44cf75bc8aef7bd.315e13e868a7f68a3ce37802b2f6312d->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf2/3a80c419c6b94682b44cf75bc8aef7bd-top 2023-04-12 02:54:10,298 DEBUG [StoreOpener-291734c00d212fd203b04cf8872d4649-1] regionserver.HStore(539): loaded hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/291734c00d212fd203b04cf8872d4649/cf2/babf83b74cec452290ff19cd45eb4713.315e13e868a7f68a3ce37802b2f6312d->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf2/babf83b74cec452290ff19cd45eb4713-top 2023-04-12 02:54:10,302 DEBUG [StoreOpener-291734c00d212fd203b04cf8872d4649-1] regionserver.HStore(539): loaded hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/291734c00d212fd203b04cf8872d4649/cf2/d253c9057ec44edbad8b758da6e18d8e.315e13e868a7f68a3ce37802b2f6312d->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf2/d253c9057ec44edbad8b758da6e18d8e-top 2023-04-12 02:54:10,309 DEBUG [StoreOpener-291734c00d212fd203b04cf8872d4649-1] regionserver.HStore(539): loaded hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/291734c00d212fd203b04cf8872d4649/cf2/d5ab6ea45d1b4298ae7d38df0de5a43f.315e13e868a7f68a3ce37802b2f6312d->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf2/d5ab6ea45d1b4298ae7d38df0de5a43f-top 2023-04-12 02:54:10,313 DEBUG [StoreOpener-291734c00d212fd203b04cf8872d4649-1] regionserver.HStore(539): loaded hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/291734c00d212fd203b04cf8872d4649/cf2/f13ea6a8012d44709becae664369d2d0.315e13e868a7f68a3ce37802b2f6312d->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf2/f13ea6a8012d44709becae664369d2d0-top 2023-04-12 02:54:10,313 INFO [StoreOpener-291734c00d212fd203b04cf8872d4649-1] regionserver.HStore(310): Store=291734c00d212fd203b04cf8872d4649/cf2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-12 02:54:10,314 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/291734c00d212fd203b04cf8872d4649 2023-04-12 02:54:10,315 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/291734c00d212fd203b04cf8872d4649 2023-04-12 02:54:10,318 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testSplitWithoutPONR descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-04-12 02:54:10,319 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 291734c00d212fd203b04cf8872d4649 2023-04-12 02:54:10,320 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 291734c00d212fd203b04cf8872d4649; next sequenceid=86; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=69982624, jitterRate=0.04282236099243164}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-04-12 02:54:10,320 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 291734c00d212fd203b04cf8872d4649: 2023-04-12 02:54:10,322 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testSplitWithoutPONR,41,1681268044634.291734c00d212fd203b04cf8872d4649., pid=34, masterSystemTime=1681268049955 2023-04-12 02:54:10,322 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: Opening Region; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 02:54:10,323 DEBUG [RS:0;jenkins-hbase4:34579-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-12 02:54:10,323 DEBUG [RS:0;jenkins-hbase4:34579-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-12 02:54:10,323 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: Opening Region; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 02:54:10,323 INFO [RS:0;jenkins-hbase4:34579-shortCompactions-0] regionserver.HStore(1898): Keeping/Overriding Compaction request priority to -2147482648 for CF cf1 since it belongs to recently split daughter region testSplitWithoutPONR,41,1681268044634.291734c00d212fd203b04cf8872d4649. 2023-04-12 02:54:10,323 INFO [RS:0;jenkins-hbase4:34579-longCompactions-0] regionserver.HStore(1898): Keeping/Overriding Compaction request priority to -2147482648 for CF cf2 since it belongs to recently split daughter region testSplitWithoutPONR,41,1681268044634.291734c00d212fd203b04cf8872d4649. 2023-04-12 02:54:10,323 DEBUG [RS:0;jenkins-hbase4:34579-longCompactions-0] regionserver.HStore(1912): 291734c00d212fd203b04cf8872d4649/cf2 is initiating minor compaction (all files) 2023-04-12 02:54:10,323 INFO [RS:0;jenkins-hbase4:34579-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 291734c00d212fd203b04cf8872d4649/cf2 in testSplitWithoutPONR,41,1681268044634.291734c00d212fd203b04cf8872d4649. 2023-04-12 02:54:10,323 DEBUG [RS:0;jenkins-hbase4:34579-shortCompactions-0] regionserver.HStore(1912): 291734c00d212fd203b04cf8872d4649/cf1 is initiating minor compaction (all files) 2023-04-12 02:54:10,324 INFO [RS:0;jenkins-hbase4:34579-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 291734c00d212fd203b04cf8872d4649/cf1 in testSplitWithoutPONR,41,1681268044634.291734c00d212fd203b04cf8872d4649. 2023-04-12 02:54:10,324 INFO [RS:0;jenkins-hbase4:34579-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/291734c00d212fd203b04cf8872d4649/cf2/0b28a4ef23e442f38cfdf4760e6f9480.315e13e868a7f68a3ce37802b2f6312d->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf2/0b28a4ef23e442f38cfdf4760e6f9480-top, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/291734c00d212fd203b04cf8872d4649/cf2/2540a68ffbdf4440b5ff30ca6a36fada.315e13e868a7f68a3ce37802b2f6312d->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf2/2540a68ffbdf4440b5ff30ca6a36fada-top, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/291734c00d212fd203b04cf8872d4649/cf2/babf83b74cec452290ff19cd45eb4713.315e13e868a7f68a3ce37802b2f6312d->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf2/babf83b74cec452290ff19cd45eb4713-top, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/291734c00d212fd203b04cf8872d4649/cf2/3a80c419c6b94682b44cf75bc8aef7bd.315e13e868a7f68a3ce37802b2f6312d->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf2/3a80c419c6b94682b44cf75bc8aef7bd-top, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/291734c00d212fd203b04cf8872d4649/cf2/f13ea6a8012d44709becae664369d2d0.315e13e868a7f68a3ce37802b2f6312d->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf2/f13ea6a8012d44709becae664369d2d0-top, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/291734c00d212fd203b04cf8872d4649/cf2/d5ab6ea45d1b4298ae7d38df0de5a43f.315e13e868a7f68a3ce37802b2f6312d->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf2/d5ab6ea45d1b4298ae7d38df0de5a43f-top, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/291734c00d212fd203b04cf8872d4649/cf2/d253c9057ec44edbad8b758da6e18d8e.315e13e868a7f68a3ce37802b2f6312d->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf2/d253c9057ec44edbad8b758da6e18d8e-top] into tmpdir=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/291734c00d212fd203b04cf8872d4649/.tmp, totalSize=34.9 K 2023-04-12 02:54:10,324 INFO [RS:0;jenkins-hbase4:34579-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/291734c00d212fd203b04cf8872d4649/cf1/1cabdcde78c14f6a8ec70790c1ae4cc4.315e13e868a7f68a3ce37802b2f6312d->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf1/1cabdcde78c14f6a8ec70790c1ae4cc4-top, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/291734c00d212fd203b04cf8872d4649/cf1/2fd181776c3749c18358aebdfc597014.315e13e868a7f68a3ce37802b2f6312d->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf1/2fd181776c3749c18358aebdfc597014-top, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/291734c00d212fd203b04cf8872d4649/cf1/55bdd10dae634a008e897b8e86048f3c.315e13e868a7f68a3ce37802b2f6312d->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf1/55bdd10dae634a008e897b8e86048f3c-top, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/291734c00d212fd203b04cf8872d4649/cf1/7e0f8c9c61e247ecbfc4c27206fae340.315e13e868a7f68a3ce37802b2f6312d->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf1/7e0f8c9c61e247ecbfc4c27206fae340-top, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/291734c00d212fd203b04cf8872d4649/cf1/2ca08ea408834a75bc73ef0d3d78fa73.315e13e868a7f68a3ce37802b2f6312d->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf1/2ca08ea408834a75bc73ef0d3d78fa73-top, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/291734c00d212fd203b04cf8872d4649/cf1/713becfd985e40a8892fbc1f6fe6bdb4.315e13e868a7f68a3ce37802b2f6312d->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf1/713becfd985e40a8892fbc1f6fe6bdb4-top, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/291734c00d212fd203b04cf8872d4649/cf1/0deaf4065ec441859d88faeb52cfa266.315e13e868a7f68a3ce37802b2f6312d->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf1/0deaf4065ec441859d88faeb52cfa266-top] into tmpdir=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/291734c00d212fd203b04cf8872d4649/.tmp, totalSize=34.9 K 2023-04-12 02:54:10,326 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testSplitWithoutPONR,41,1681268044634.291734c00d212fd203b04cf8872d4649. 2023-04-12 02:54:10,326 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testSplitWithoutPONR,41,1681268044634.291734c00d212fd203b04cf8872d4649. 2023-04-12 02:54:10,327 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=32 updating hbase:meta row=291734c00d212fd203b04cf8872d4649, regionState=OPEN, openSeqNum=86, regionLocation=jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:10,327 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testSplitWithoutPONR,41,1681268044634.291734c00d212fd203b04cf8872d4649.","families":{"info":[{"qualifier":"regioninfo","vlen":56,"tag":[],"timestamp":"1681268050327"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1681268050327"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681268050327"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681268050327"}]},"ts":"1681268050327"} 2023-04-12 02:54:10,331 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=34, resume processing ppid=32 2023-04-12 02:54:10,331 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=34, ppid=32, state=SUCCESS; OpenRegionProcedure 291734c00d212fd203b04cf8872d4649, server=jenkins-hbase4.apache.org,34579,1681268031719 in 516 msec 2023-04-12 02:54:10,335 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=32, resume processing ppid=28 2023-04-12 02:54:10,335 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=32, ppid=28, state=SUCCESS; TransitRegionStateProcedure table=testSplitWithoutPONR, region=291734c00d212fd203b04cf8872d4649, ASSIGN in 692 msec 2023-04-12 02:54:10,336 DEBUG [RS:0;jenkins-hbase4:34579-longCompactions-0] compactions.Compactor(207): Compacting 0b28a4ef23e442f38cfdf4760e6f9480.315e13e868a7f68a3ce37802b2f6312d, keycount=1, bloomtype=ROW, size=4.8 K, encoding=NONE, compression=NONE, seqNum=7, earliestPutTs=1681268042050 2023-04-12 02:54:10,336 DEBUG [RS:0;jenkins-hbase4:34579-shortCompactions-0] compactions.Compactor(207): Compacting 1cabdcde78c14f6a8ec70790c1ae4cc4.315e13e868a7f68a3ce37802b2f6312d, keycount=1, bloomtype=ROW, size=4.8 K, encoding=NONE, compression=NONE, seqNum=7, earliestPutTs=1681268042050 2023-04-12 02:54:10,337 DEBUG [RS:0;jenkins-hbase4:34579-longCompactions-0] compactions.Compactor(207): Compacting 2540a68ffbdf4440b5ff30ca6a36fada.315e13e868a7f68a3ce37802b2f6312d, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=20, earliestPutTs=1681268042410 2023-04-12 02:54:10,338 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=28, state=SUCCESS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=315e13e868a7f68a3ce37802b2f6312d, daughterA=4e1092f3c00484f8f09c730861225d43, daughterB=291734c00d212fd203b04cf8872d4649 in 5.7010 sec 2023-04-12 02:54:10,338 DEBUG [RS:0;jenkins-hbase4:34579-shortCompactions-0] compactions.Compactor(207): Compacting 2fd181776c3749c18358aebdfc597014.315e13e868a7f68a3ce37802b2f6312d, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=20, earliestPutTs=1681268042410 2023-04-12 02:54:10,338 DEBUG [RS:0;jenkins-hbase4:34579-longCompactions-0] compactions.Compactor(207): Compacting babf83b74cec452290ff19cd45eb4713.315e13e868a7f68a3ce37802b2f6312d, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=33, earliestPutTs=1681268042730 2023-04-12 02:54:10,339 DEBUG [RS:0;jenkins-hbase4:34579-shortCompactions-0] compactions.Compactor(207): Compacting 55bdd10dae634a008e897b8e86048f3c.315e13e868a7f68a3ce37802b2f6312d, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=33, earliestPutTs=1681268042730 2023-04-12 02:54:10,339 DEBUG [RS:0;jenkins-hbase4:34579-longCompactions-0] compactions.Compactor(207): Compacting 3a80c419c6b94682b44cf75bc8aef7bd.315e13e868a7f68a3ce37802b2f6312d, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=46, earliestPutTs=1681268043069 2023-04-12 02:54:10,340 DEBUG [RS:0;jenkins-hbase4:34579-shortCompactions-0] compactions.Compactor(207): Compacting 7e0f8c9c61e247ecbfc4c27206fae340.315e13e868a7f68a3ce37802b2f6312d, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=46, earliestPutTs=1681268043069 2023-04-12 02:54:10,340 DEBUG [RS:0;jenkins-hbase4:34579-longCompactions-0] compactions.Compactor(207): Compacting f13ea6a8012d44709becae664369d2d0.315e13e868a7f68a3ce37802b2f6312d, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=59, earliestPutTs=1681268043360 2023-04-12 02:54:10,341 DEBUG [RS:0;jenkins-hbase4:34579-longCompactions-0] compactions.Compactor(207): Compacting d5ab6ea45d1b4298ae7d38df0de5a43f.315e13e868a7f68a3ce37802b2f6312d, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=72, earliestPutTs=1681268043630 2023-04-12 02:54:10,341 DEBUG [RS:0;jenkins-hbase4:34579-shortCompactions-0] compactions.Compactor(207): Compacting 2ca08ea408834a75bc73ef0d3d78fa73.315e13e868a7f68a3ce37802b2f6312d, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=59, earliestPutTs=1681268043360 2023-04-12 02:54:10,343 DEBUG [RS:0;jenkins-hbase4:34579-longCompactions-0] compactions.Compactor(207): Compacting d253c9057ec44edbad8b758da6e18d8e.315e13e868a7f68a3ce37802b2f6312d, keycount=4, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=83, earliestPutTs=1681268044369 2023-04-12 02:54:10,343 DEBUG [RS:0;jenkins-hbase4:34579-shortCompactions-0] compactions.Compactor(207): Compacting 713becfd985e40a8892fbc1f6fe6bdb4.315e13e868a7f68a3ce37802b2f6312d, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=72, earliestPutTs=1681268043630 2023-04-12 02:54:10,345 DEBUG [RS:0;jenkins-hbase4:34579-shortCompactions-0] compactions.Compactor(207): Compacting 0deaf4065ec441859d88faeb52cfa266.315e13e868a7f68a3ce37802b2f6312d, keycount=4, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=83, earliestPutTs=1681268044369 2023-04-12 02:54:10,368 INFO [RS:0;jenkins-hbase4:34579-longCompactions-0] throttle.PressureAwareThroughputController(145): 291734c00d212fd203b04cf8872d4649#cf2#compaction#34 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 02:54:10,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34579] ipc.CallRunner(144): callId: 241 service: ClientService methodName: Scan size: 143 connection: 172.31.14.131:49302 deadline: 1681268110392, exception=org.apache.hadoop.hbase.NotServingRegionException: testSplitWithoutPONR,,1681268041279.315e13e868a7f68a3ce37802b2f6312d. is not online on jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:10,397 INFO [RS:0;jenkins-hbase4:34579-shortCompactions-0] throttle.PressureAwareThroughputController(145): 291734c00d212fd203b04cf8872d4649#cf1#compaction#35 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 02:54:10,448 DEBUG [RS:0;jenkins-hbase4:34579-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/291734c00d212fd203b04cf8872d4649/.tmp/cf2/8c55c3693a994e108164d71a69ad36d6 as hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/291734c00d212fd203b04cf8872d4649/cf2/8c55c3693a994e108164d71a69ad36d6 2023-04-12 02:54:10,453 DEBUG [RS:0;jenkins-hbase4:34579-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/291734c00d212fd203b04cf8872d4649/.tmp/cf1/7a1a6c7dd03543e880b73d3a48fa09bc as hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/291734c00d212fd203b04cf8872d4649/cf1/7a1a6c7dd03543e880b73d3a48fa09bc 2023-04-12 02:54:10,461 INFO [RS:0;jenkins-hbase4:34579-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 291734c00d212fd203b04cf8872d4649/cf2 of 291734c00d212fd203b04cf8872d4649 into 8c55c3693a994e108164d71a69ad36d6(size=6.1 K), total size for store is 6.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 02:54:10,461 DEBUG [RS:0;jenkins-hbase4:34579-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 291734c00d212fd203b04cf8872d4649: 2023-04-12 02:54:10,461 INFO [RS:0;jenkins-hbase4:34579-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=testSplitWithoutPONR,41,1681268044634.291734c00d212fd203b04cf8872d4649., storeName=291734c00d212fd203b04cf8872d4649/cf2, priority=9, startTime=1681268050323; duration=0sec 2023-04-12 02:54:10,462 DEBUG [RS:0;jenkins-hbase4:34579-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 02:54:10,478 INFO [RS:0;jenkins-hbase4:34579-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 291734c00d212fd203b04cf8872d4649/cf1 of 291734c00d212fd203b04cf8872d4649 into 7a1a6c7dd03543e880b73d3a48fa09bc(size=6.1 K), total size for store is 6.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 02:54:10,478 DEBUG [RS:0;jenkins-hbase4:34579-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 291734c00d212fd203b04cf8872d4649: 2023-04-12 02:54:10,479 INFO [RS:0;jenkins-hbase4:34579-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=testSplitWithoutPONR,41,1681268044634.291734c00d212fd203b04cf8872d4649., storeName=291734c00d212fd203b04cf8872d4649/cf1, priority=9, startTime=1681268050322; duration=0sec 2023-04-12 02:54:10,479 DEBUG [RS:0;jenkins-hbase4:34579-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 02:54:10,511 DEBUG [Listener at localhost/46667] client.ClientScanner(240): Advancing internal scanner to startKey at '41', inclusive 2023-04-12 02:54:10,544 INFO [Listener at localhost/46667] assignment.TestSplitTableRegionProcedure(570): Row Count = 60 2023-04-12 02:54:10,547 DEBUG [Listener at localhost/46667] client.ClientScanner(240): Advancing internal scanner to startKey at '41', inclusive 2023-04-12 02:54:10,572 WARN [Listener at localhost/46667] procedure2.ProcedureTestingUtility(193): Set Kill before store update to: false 2023-04-12 02:54:10,579 INFO [Listener at localhost/46667] client.HBaseAdmin$15(890): Started disable of testSplitWithoutPONR 2023-04-12 02:54:10,579 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable testSplitWithoutPONR 2023-04-12 02:54:10,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] procedure2.ProcedureExecutor(1029): Stored pid=35, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testSplitWithoutPONR 2023-04-12 02:54:10,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-12 02:54:10,587 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testSplitWithoutPONR","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681268050587"}]},"ts":"1681268050587"} 2023-04-12 02:54:10,589 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testSplitWithoutPONR, state=DISABLING in hbase:meta 2023-04-12 02:54:10,590 INFO [PEWorker-1] procedure.DisableTableProcedure(293): Set testSplitWithoutPONR to state=DISABLING 2023-04-12 02:54:10,593 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitWithoutPONR, region=4e1092f3c00484f8f09c730861225d43, UNASSIGN}, {pid=37, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitWithoutPONR, region=291734c00d212fd203b04cf8872d4649, UNASSIGN}] 2023-04-12 02:54:10,595 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=37, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitWithoutPONR, region=291734c00d212fd203b04cf8872d4649, UNASSIGN 2023-04-12 02:54:10,596 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=37 updating hbase:meta row=291734c00d212fd203b04cf8872d4649, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:10,596 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitWithoutPONR,41,1681268044634.291734c00d212fd203b04cf8872d4649.","families":{"info":[{"qualifier":"regioninfo","vlen":56,"tag":[],"timestamp":"1681268050596"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681268050596"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681268050596"}]},"ts":"1681268050596"} 2023-04-12 02:54:10,598 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=38, ppid=37, state=RUNNABLE; CloseRegionProcedure 291734c00d212fd203b04cf8872d4649, server=jenkins-hbase4.apache.org,34579,1681268031719}] 2023-04-12 02:54:10,603 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitWithoutPONR, region=4e1092f3c00484f8f09c730861225d43, UNASSIGN 2023-04-12 02:54:10,604 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=36 updating hbase:meta row=4e1092f3c00484f8f09c730861225d43, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:10,604 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitWithoutPONR,,1681268044634.4e1092f3c00484f8f09c730861225d43.","families":{"info":[{"qualifier":"regioninfo","vlen":56,"tag":[],"timestamp":"1681268050604"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681268050604"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681268050604"}]},"ts":"1681268050604"} 2023-04-12 02:54:10,605 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=39, ppid=36, state=RUNNABLE; CloseRegionProcedure 4e1092f3c00484f8f09c730861225d43, server=jenkins-hbase4.apache.org,34579,1681268031719}] 2023-04-12 02:54:10,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-12 02:54:10,752 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 291734c00d212fd203b04cf8872d4649 2023-04-12 02:54:10,752 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 291734c00d212fd203b04cf8872d4649, disabling compactions & flushes 2023-04-12 02:54:10,752 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region testSplitWithoutPONR,41,1681268044634.291734c00d212fd203b04cf8872d4649. 2023-04-12 02:54:10,752 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testSplitWithoutPONR,41,1681268044634.291734c00d212fd203b04cf8872d4649. 2023-04-12 02:54:10,753 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on testSplitWithoutPONR,41,1681268044634.291734c00d212fd203b04cf8872d4649. after waiting 0 ms 2023-04-12 02:54:10,753 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region testSplitWithoutPONR,41,1681268044634.291734c00d212fd203b04cf8872d4649. 2023-04-12 02:54:10,769 DEBUG [StoreCloser-testSplitWithoutPONR,41,1681268044634.291734c00d212fd203b04cf8872d4649.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/291734c00d212fd203b04cf8872d4649/cf1/1cabdcde78c14f6a8ec70790c1ae4cc4.315e13e868a7f68a3ce37802b2f6312d->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf1/1cabdcde78c14f6a8ec70790c1ae4cc4-top, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/291734c00d212fd203b04cf8872d4649/cf1/2fd181776c3749c18358aebdfc597014.315e13e868a7f68a3ce37802b2f6312d->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf1/2fd181776c3749c18358aebdfc597014-top, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/291734c00d212fd203b04cf8872d4649/cf1/55bdd10dae634a008e897b8e86048f3c.315e13e868a7f68a3ce37802b2f6312d->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf1/55bdd10dae634a008e897b8e86048f3c-top, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/291734c00d212fd203b04cf8872d4649/cf1/7e0f8c9c61e247ecbfc4c27206fae340.315e13e868a7f68a3ce37802b2f6312d->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf1/7e0f8c9c61e247ecbfc4c27206fae340-top, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/291734c00d212fd203b04cf8872d4649/cf1/2ca08ea408834a75bc73ef0d3d78fa73.315e13e868a7f68a3ce37802b2f6312d->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf1/2ca08ea408834a75bc73ef0d3d78fa73-top, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/291734c00d212fd203b04cf8872d4649/cf1/713becfd985e40a8892fbc1f6fe6bdb4.315e13e868a7f68a3ce37802b2f6312d->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf1/713becfd985e40a8892fbc1f6fe6bdb4-top, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/291734c00d212fd203b04cf8872d4649/cf1/0deaf4065ec441859d88faeb52cfa266.315e13e868a7f68a3ce37802b2f6312d->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf1/0deaf4065ec441859d88faeb52cfa266-top] to archive 2023-04-12 02:54:10,770 DEBUG [StoreCloser-testSplitWithoutPONR,41,1681268044634.291734c00d212fd203b04cf8872d4649.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-12 02:54:10,774 DEBUG [StoreCloser-testSplitWithoutPONR,41,1681268044634.291734c00d212fd203b04cf8872d4649.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/291734c00d212fd203b04cf8872d4649/cf1/1cabdcde78c14f6a8ec70790c1ae4cc4.315e13e868a7f68a3ce37802b2f6312d to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitWithoutPONR/291734c00d212fd203b04cf8872d4649/cf1/1cabdcde78c14f6a8ec70790c1ae4cc4.315e13e868a7f68a3ce37802b2f6312d 2023-04-12 02:54:10,784 DEBUG [StoreCloser-testSplitWithoutPONR,41,1681268044634.291734c00d212fd203b04cf8872d4649.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/291734c00d212fd203b04cf8872d4649/cf1/2fd181776c3749c18358aebdfc597014.315e13e868a7f68a3ce37802b2f6312d to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitWithoutPONR/291734c00d212fd203b04cf8872d4649/cf1/2fd181776c3749c18358aebdfc597014.315e13e868a7f68a3ce37802b2f6312d 2023-04-12 02:54:10,788 DEBUG [StoreCloser-testSplitWithoutPONR,41,1681268044634.291734c00d212fd203b04cf8872d4649.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/291734c00d212fd203b04cf8872d4649/cf1/55bdd10dae634a008e897b8e86048f3c.315e13e868a7f68a3ce37802b2f6312d to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitWithoutPONR/291734c00d212fd203b04cf8872d4649/cf1/55bdd10dae634a008e897b8e86048f3c.315e13e868a7f68a3ce37802b2f6312d 2023-04-12 02:54:10,790 DEBUG [StoreCloser-testSplitWithoutPONR,41,1681268044634.291734c00d212fd203b04cf8872d4649.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/291734c00d212fd203b04cf8872d4649/cf1/7e0f8c9c61e247ecbfc4c27206fae340.315e13e868a7f68a3ce37802b2f6312d to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitWithoutPONR/291734c00d212fd203b04cf8872d4649/cf1/7e0f8c9c61e247ecbfc4c27206fae340.315e13e868a7f68a3ce37802b2f6312d 2023-04-12 02:54:10,792 DEBUG [StoreCloser-testSplitWithoutPONR,41,1681268044634.291734c00d212fd203b04cf8872d4649.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/291734c00d212fd203b04cf8872d4649/cf1/2ca08ea408834a75bc73ef0d3d78fa73.315e13e868a7f68a3ce37802b2f6312d to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitWithoutPONR/291734c00d212fd203b04cf8872d4649/cf1/2ca08ea408834a75bc73ef0d3d78fa73.315e13e868a7f68a3ce37802b2f6312d 2023-04-12 02:54:10,794 DEBUG [StoreCloser-testSplitWithoutPONR,41,1681268044634.291734c00d212fd203b04cf8872d4649.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/291734c00d212fd203b04cf8872d4649/cf1/713becfd985e40a8892fbc1f6fe6bdb4.315e13e868a7f68a3ce37802b2f6312d to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitWithoutPONR/291734c00d212fd203b04cf8872d4649/cf1/713becfd985e40a8892fbc1f6fe6bdb4.315e13e868a7f68a3ce37802b2f6312d 2023-04-12 02:54:10,795 DEBUG [StoreCloser-testSplitWithoutPONR,41,1681268044634.291734c00d212fd203b04cf8872d4649.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/291734c00d212fd203b04cf8872d4649/cf1/0deaf4065ec441859d88faeb52cfa266.315e13e868a7f68a3ce37802b2f6312d to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitWithoutPONR/291734c00d212fd203b04cf8872d4649/cf1/0deaf4065ec441859d88faeb52cfa266.315e13e868a7f68a3ce37802b2f6312d 2023-04-12 02:54:10,802 DEBUG [StoreCloser-testSplitWithoutPONR,41,1681268044634.291734c00d212fd203b04cf8872d4649.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/291734c00d212fd203b04cf8872d4649/cf2/0b28a4ef23e442f38cfdf4760e6f9480.315e13e868a7f68a3ce37802b2f6312d->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf2/0b28a4ef23e442f38cfdf4760e6f9480-top, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/291734c00d212fd203b04cf8872d4649/cf2/2540a68ffbdf4440b5ff30ca6a36fada.315e13e868a7f68a3ce37802b2f6312d->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf2/2540a68ffbdf4440b5ff30ca6a36fada-top, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/291734c00d212fd203b04cf8872d4649/cf2/babf83b74cec452290ff19cd45eb4713.315e13e868a7f68a3ce37802b2f6312d->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf2/babf83b74cec452290ff19cd45eb4713-top, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/291734c00d212fd203b04cf8872d4649/cf2/3a80c419c6b94682b44cf75bc8aef7bd.315e13e868a7f68a3ce37802b2f6312d->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf2/3a80c419c6b94682b44cf75bc8aef7bd-top, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/291734c00d212fd203b04cf8872d4649/cf2/f13ea6a8012d44709becae664369d2d0.315e13e868a7f68a3ce37802b2f6312d->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf2/f13ea6a8012d44709becae664369d2d0-top, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/291734c00d212fd203b04cf8872d4649/cf2/d5ab6ea45d1b4298ae7d38df0de5a43f.315e13e868a7f68a3ce37802b2f6312d->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf2/d5ab6ea45d1b4298ae7d38df0de5a43f-top, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/291734c00d212fd203b04cf8872d4649/cf2/d253c9057ec44edbad8b758da6e18d8e.315e13e868a7f68a3ce37802b2f6312d->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf2/d253c9057ec44edbad8b758da6e18d8e-top] to archive 2023-04-12 02:54:10,803 DEBUG [StoreCloser-testSplitWithoutPONR,41,1681268044634.291734c00d212fd203b04cf8872d4649.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-12 02:54:10,805 DEBUG [StoreCloser-testSplitWithoutPONR,41,1681268044634.291734c00d212fd203b04cf8872d4649.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/291734c00d212fd203b04cf8872d4649/cf2/0b28a4ef23e442f38cfdf4760e6f9480.315e13e868a7f68a3ce37802b2f6312d to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitWithoutPONR/291734c00d212fd203b04cf8872d4649/cf2/0b28a4ef23e442f38cfdf4760e6f9480.315e13e868a7f68a3ce37802b2f6312d 2023-04-12 02:54:10,807 DEBUG [StoreCloser-testSplitWithoutPONR,41,1681268044634.291734c00d212fd203b04cf8872d4649.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/291734c00d212fd203b04cf8872d4649/cf2/2540a68ffbdf4440b5ff30ca6a36fada.315e13e868a7f68a3ce37802b2f6312d to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitWithoutPONR/291734c00d212fd203b04cf8872d4649/cf2/2540a68ffbdf4440b5ff30ca6a36fada.315e13e868a7f68a3ce37802b2f6312d 2023-04-12 02:54:10,810 DEBUG [StoreCloser-testSplitWithoutPONR,41,1681268044634.291734c00d212fd203b04cf8872d4649.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/291734c00d212fd203b04cf8872d4649/cf2/babf83b74cec452290ff19cd45eb4713.315e13e868a7f68a3ce37802b2f6312d to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitWithoutPONR/291734c00d212fd203b04cf8872d4649/cf2/babf83b74cec452290ff19cd45eb4713.315e13e868a7f68a3ce37802b2f6312d 2023-04-12 02:54:10,812 DEBUG [StoreCloser-testSplitWithoutPONR,41,1681268044634.291734c00d212fd203b04cf8872d4649.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/291734c00d212fd203b04cf8872d4649/cf2/3a80c419c6b94682b44cf75bc8aef7bd.315e13e868a7f68a3ce37802b2f6312d to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitWithoutPONR/291734c00d212fd203b04cf8872d4649/cf2/3a80c419c6b94682b44cf75bc8aef7bd.315e13e868a7f68a3ce37802b2f6312d 2023-04-12 02:54:10,814 DEBUG [StoreCloser-testSplitWithoutPONR,41,1681268044634.291734c00d212fd203b04cf8872d4649.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/291734c00d212fd203b04cf8872d4649/cf2/f13ea6a8012d44709becae664369d2d0.315e13e868a7f68a3ce37802b2f6312d to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitWithoutPONR/291734c00d212fd203b04cf8872d4649/cf2/f13ea6a8012d44709becae664369d2d0.315e13e868a7f68a3ce37802b2f6312d 2023-04-12 02:54:10,815 DEBUG [StoreCloser-testSplitWithoutPONR,41,1681268044634.291734c00d212fd203b04cf8872d4649.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/291734c00d212fd203b04cf8872d4649/cf2/d5ab6ea45d1b4298ae7d38df0de5a43f.315e13e868a7f68a3ce37802b2f6312d to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitWithoutPONR/291734c00d212fd203b04cf8872d4649/cf2/d5ab6ea45d1b4298ae7d38df0de5a43f.315e13e868a7f68a3ce37802b2f6312d 2023-04-12 02:54:10,817 DEBUG [StoreCloser-testSplitWithoutPONR,41,1681268044634.291734c00d212fd203b04cf8872d4649.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/291734c00d212fd203b04cf8872d4649/cf2/d253c9057ec44edbad8b758da6e18d8e.315e13e868a7f68a3ce37802b2f6312d to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitWithoutPONR/291734c00d212fd203b04cf8872d4649/cf2/d253c9057ec44edbad8b758da6e18d8e.315e13e868a7f68a3ce37802b2f6312d 2023-04-12 02:54:10,821 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/291734c00d212fd203b04cf8872d4649/recovered.edits/90.seqid, newMaxSeqId=90, maxSeqId=85 2023-04-12 02:54:10,822 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro 2023-04-12 02:54:10,822 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed testSplitWithoutPONR,41,1681268044634.291734c00d212fd203b04cf8872d4649. 2023-04-12 02:54:10,822 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 291734c00d212fd203b04cf8872d4649: 2023-04-12 02:54:10,824 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 291734c00d212fd203b04cf8872d4649 2023-04-12 02:54:10,824 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 4e1092f3c00484f8f09c730861225d43 2023-04-12 02:54:10,824 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 4e1092f3c00484f8f09c730861225d43, disabling compactions & flushes 2023-04-12 02:54:10,824 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region testSplitWithoutPONR,,1681268044634.4e1092f3c00484f8f09c730861225d43. 2023-04-12 02:54:10,824 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testSplitWithoutPONR,,1681268044634.4e1092f3c00484f8f09c730861225d43. 2023-04-12 02:54:10,824 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on testSplitWithoutPONR,,1681268044634.4e1092f3c00484f8f09c730861225d43. after waiting 0 ms 2023-04-12 02:54:10,824 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region testSplitWithoutPONR,,1681268044634.4e1092f3c00484f8f09c730861225d43. 2023-04-12 02:54:10,825 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=37 updating hbase:meta row=291734c00d212fd203b04cf8872d4649, regionState=CLOSED 2023-04-12 02:54:10,825 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testSplitWithoutPONR,41,1681268044634.291734c00d212fd203b04cf8872d4649.","families":{"info":[{"qualifier":"regioninfo","vlen":56,"tag":[],"timestamp":"1681268050825"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681268050825"}]},"ts":"1681268050825"} 2023-04-12 02:54:10,829 DEBUG [StoreCloser-testSplitWithoutPONR,,1681268044634.4e1092f3c00484f8f09c730861225d43.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/4e1092f3c00484f8f09c730861225d43/cf1/1cabdcde78c14f6a8ec70790c1ae4cc4.315e13e868a7f68a3ce37802b2f6312d->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf1/1cabdcde78c14f6a8ec70790c1ae4cc4-bottom, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/4e1092f3c00484f8f09c730861225d43/cf1/2fd181776c3749c18358aebdfc597014.315e13e868a7f68a3ce37802b2f6312d->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf1/2fd181776c3749c18358aebdfc597014-bottom, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/4e1092f3c00484f8f09c730861225d43/cf1/55bdd10dae634a008e897b8e86048f3c.315e13e868a7f68a3ce37802b2f6312d->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf1/55bdd10dae634a008e897b8e86048f3c-bottom, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/4e1092f3c00484f8f09c730861225d43/cf1/7e0f8c9c61e247ecbfc4c27206fae340.315e13e868a7f68a3ce37802b2f6312d->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf1/7e0f8c9c61e247ecbfc4c27206fae340-bottom, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/4e1092f3c00484f8f09c730861225d43/cf1/2ca08ea408834a75bc73ef0d3d78fa73.315e13e868a7f68a3ce37802b2f6312d->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf1/2ca08ea408834a75bc73ef0d3d78fa73-bottom, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/4e1092f3c00484f8f09c730861225d43/cf1/713becfd985e40a8892fbc1f6fe6bdb4.315e13e868a7f68a3ce37802b2f6312d->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf1/713becfd985e40a8892fbc1f6fe6bdb4-bottom, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/4e1092f3c00484f8f09c730861225d43/cf1/0deaf4065ec441859d88faeb52cfa266.315e13e868a7f68a3ce37802b2f6312d->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf1/0deaf4065ec441859d88faeb52cfa266-bottom] to archive 2023-04-12 02:54:10,830 DEBUG [StoreCloser-testSplitWithoutPONR,,1681268044634.4e1092f3c00484f8f09c730861225d43.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-12 02:54:10,833 DEBUG [StoreCloser-testSplitWithoutPONR,,1681268044634.4e1092f3c00484f8f09c730861225d43.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/4e1092f3c00484f8f09c730861225d43/cf1/1cabdcde78c14f6a8ec70790c1ae4cc4.315e13e868a7f68a3ce37802b2f6312d to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitWithoutPONR/4e1092f3c00484f8f09c730861225d43/cf1/1cabdcde78c14f6a8ec70790c1ae4cc4.315e13e868a7f68a3ce37802b2f6312d 2023-04-12 02:54:10,838 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=38, resume processing ppid=37 2023-04-12 02:54:10,838 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=38, ppid=37, state=SUCCESS; CloseRegionProcedure 291734c00d212fd203b04cf8872d4649, server=jenkins-hbase4.apache.org,34579,1681268031719 in 229 msec 2023-04-12 02:54:10,839 DEBUG [StoreCloser-testSplitWithoutPONR,,1681268044634.4e1092f3c00484f8f09c730861225d43.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/4e1092f3c00484f8f09c730861225d43/cf1/2fd181776c3749c18358aebdfc597014.315e13e868a7f68a3ce37802b2f6312d to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitWithoutPONR/4e1092f3c00484f8f09c730861225d43/cf1/2fd181776c3749c18358aebdfc597014.315e13e868a7f68a3ce37802b2f6312d 2023-04-12 02:54:10,840 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=37, ppid=35, state=SUCCESS; TransitRegionStateProcedure table=testSplitWithoutPONR, region=291734c00d212fd203b04cf8872d4649, UNASSIGN in 245 msec 2023-04-12 02:54:10,842 DEBUG [StoreCloser-testSplitWithoutPONR,,1681268044634.4e1092f3c00484f8f09c730861225d43.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/4e1092f3c00484f8f09c730861225d43/cf1/55bdd10dae634a008e897b8e86048f3c.315e13e868a7f68a3ce37802b2f6312d to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitWithoutPONR/4e1092f3c00484f8f09c730861225d43/cf1/55bdd10dae634a008e897b8e86048f3c.315e13e868a7f68a3ce37802b2f6312d 2023-04-12 02:54:10,843 DEBUG [StoreCloser-testSplitWithoutPONR,,1681268044634.4e1092f3c00484f8f09c730861225d43.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/4e1092f3c00484f8f09c730861225d43/cf1/7e0f8c9c61e247ecbfc4c27206fae340.315e13e868a7f68a3ce37802b2f6312d to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitWithoutPONR/4e1092f3c00484f8f09c730861225d43/cf1/7e0f8c9c61e247ecbfc4c27206fae340.315e13e868a7f68a3ce37802b2f6312d 2023-04-12 02:54:10,845 DEBUG [StoreCloser-testSplitWithoutPONR,,1681268044634.4e1092f3c00484f8f09c730861225d43.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/4e1092f3c00484f8f09c730861225d43/cf1/2ca08ea408834a75bc73ef0d3d78fa73.315e13e868a7f68a3ce37802b2f6312d to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitWithoutPONR/4e1092f3c00484f8f09c730861225d43/cf1/2ca08ea408834a75bc73ef0d3d78fa73.315e13e868a7f68a3ce37802b2f6312d 2023-04-12 02:54:10,847 DEBUG [StoreCloser-testSplitWithoutPONR,,1681268044634.4e1092f3c00484f8f09c730861225d43.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/4e1092f3c00484f8f09c730861225d43/cf1/713becfd985e40a8892fbc1f6fe6bdb4.315e13e868a7f68a3ce37802b2f6312d to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitWithoutPONR/4e1092f3c00484f8f09c730861225d43/cf1/713becfd985e40a8892fbc1f6fe6bdb4.315e13e868a7f68a3ce37802b2f6312d 2023-04-12 02:54:10,848 DEBUG [StoreCloser-testSplitWithoutPONR,,1681268044634.4e1092f3c00484f8f09c730861225d43.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/4e1092f3c00484f8f09c730861225d43/cf1/0deaf4065ec441859d88faeb52cfa266.315e13e868a7f68a3ce37802b2f6312d to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitWithoutPONR/4e1092f3c00484f8f09c730861225d43/cf1/0deaf4065ec441859d88faeb52cfa266.315e13e868a7f68a3ce37802b2f6312d 2023-04-12 02:54:10,851 DEBUG [StoreCloser-testSplitWithoutPONR,,1681268044634.4e1092f3c00484f8f09c730861225d43.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/4e1092f3c00484f8f09c730861225d43/cf2/0b28a4ef23e442f38cfdf4760e6f9480.315e13e868a7f68a3ce37802b2f6312d->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf2/0b28a4ef23e442f38cfdf4760e6f9480-bottom, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/4e1092f3c00484f8f09c730861225d43/cf2/2540a68ffbdf4440b5ff30ca6a36fada.315e13e868a7f68a3ce37802b2f6312d->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf2/2540a68ffbdf4440b5ff30ca6a36fada-bottom, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/4e1092f3c00484f8f09c730861225d43/cf2/babf83b74cec452290ff19cd45eb4713.315e13e868a7f68a3ce37802b2f6312d->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf2/babf83b74cec452290ff19cd45eb4713-bottom, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/4e1092f3c00484f8f09c730861225d43/cf2/3a80c419c6b94682b44cf75bc8aef7bd.315e13e868a7f68a3ce37802b2f6312d->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf2/3a80c419c6b94682b44cf75bc8aef7bd-bottom, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/4e1092f3c00484f8f09c730861225d43/cf2/f13ea6a8012d44709becae664369d2d0.315e13e868a7f68a3ce37802b2f6312d->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf2/f13ea6a8012d44709becae664369d2d0-bottom, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/4e1092f3c00484f8f09c730861225d43/cf2/d5ab6ea45d1b4298ae7d38df0de5a43f.315e13e868a7f68a3ce37802b2f6312d->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf2/d5ab6ea45d1b4298ae7d38df0de5a43f-bottom, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/4e1092f3c00484f8f09c730861225d43/cf2/d253c9057ec44edbad8b758da6e18d8e.315e13e868a7f68a3ce37802b2f6312d->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf2/d253c9057ec44edbad8b758da6e18d8e-bottom] to archive 2023-04-12 02:54:10,852 DEBUG [StoreCloser-testSplitWithoutPONR,,1681268044634.4e1092f3c00484f8f09c730861225d43.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-12 02:54:10,853 DEBUG [StoreCloser-testSplitWithoutPONR,,1681268044634.4e1092f3c00484f8f09c730861225d43.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/4e1092f3c00484f8f09c730861225d43/cf2/0b28a4ef23e442f38cfdf4760e6f9480.315e13e868a7f68a3ce37802b2f6312d to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitWithoutPONR/4e1092f3c00484f8f09c730861225d43/cf2/0b28a4ef23e442f38cfdf4760e6f9480.315e13e868a7f68a3ce37802b2f6312d 2023-04-12 02:54:10,855 DEBUG [StoreCloser-testSplitWithoutPONR,,1681268044634.4e1092f3c00484f8f09c730861225d43.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/4e1092f3c00484f8f09c730861225d43/cf2/2540a68ffbdf4440b5ff30ca6a36fada.315e13e868a7f68a3ce37802b2f6312d to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitWithoutPONR/4e1092f3c00484f8f09c730861225d43/cf2/2540a68ffbdf4440b5ff30ca6a36fada.315e13e868a7f68a3ce37802b2f6312d 2023-04-12 02:54:10,857 DEBUG [StoreCloser-testSplitWithoutPONR,,1681268044634.4e1092f3c00484f8f09c730861225d43.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/4e1092f3c00484f8f09c730861225d43/cf2/babf83b74cec452290ff19cd45eb4713.315e13e868a7f68a3ce37802b2f6312d to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitWithoutPONR/4e1092f3c00484f8f09c730861225d43/cf2/babf83b74cec452290ff19cd45eb4713.315e13e868a7f68a3ce37802b2f6312d 2023-04-12 02:54:10,858 DEBUG [StoreCloser-testSplitWithoutPONR,,1681268044634.4e1092f3c00484f8f09c730861225d43.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/4e1092f3c00484f8f09c730861225d43/cf2/3a80c419c6b94682b44cf75bc8aef7bd.315e13e868a7f68a3ce37802b2f6312d to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitWithoutPONR/4e1092f3c00484f8f09c730861225d43/cf2/3a80c419c6b94682b44cf75bc8aef7bd.315e13e868a7f68a3ce37802b2f6312d 2023-04-12 02:54:10,859 DEBUG [StoreCloser-testSplitWithoutPONR,,1681268044634.4e1092f3c00484f8f09c730861225d43.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/4e1092f3c00484f8f09c730861225d43/cf2/f13ea6a8012d44709becae664369d2d0.315e13e868a7f68a3ce37802b2f6312d to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitWithoutPONR/4e1092f3c00484f8f09c730861225d43/cf2/f13ea6a8012d44709becae664369d2d0.315e13e868a7f68a3ce37802b2f6312d 2023-04-12 02:54:10,862 DEBUG [StoreCloser-testSplitWithoutPONR,,1681268044634.4e1092f3c00484f8f09c730861225d43.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/4e1092f3c00484f8f09c730861225d43/cf2/d5ab6ea45d1b4298ae7d38df0de5a43f.315e13e868a7f68a3ce37802b2f6312d to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitWithoutPONR/4e1092f3c00484f8f09c730861225d43/cf2/d5ab6ea45d1b4298ae7d38df0de5a43f.315e13e868a7f68a3ce37802b2f6312d 2023-04-12 02:54:10,863 DEBUG [StoreCloser-testSplitWithoutPONR,,1681268044634.4e1092f3c00484f8f09c730861225d43.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/4e1092f3c00484f8f09c730861225d43/cf2/d253c9057ec44edbad8b758da6e18d8e.315e13e868a7f68a3ce37802b2f6312d to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitWithoutPONR/4e1092f3c00484f8f09c730861225d43/cf2/d253c9057ec44edbad8b758da6e18d8e.315e13e868a7f68a3ce37802b2f6312d 2023-04-12 02:54:10,868 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitWithoutPONR/4e1092f3c00484f8f09c730861225d43/recovered.edits/90.seqid, newMaxSeqId=90, maxSeqId=85 2023-04-12 02:54:10,869 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro 2023-04-12 02:54:10,869 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed testSplitWithoutPONR,,1681268044634.4e1092f3c00484f8f09c730861225d43. 2023-04-12 02:54:10,869 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 4e1092f3c00484f8f09c730861225d43: 2023-04-12 02:54:10,871 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 4e1092f3c00484f8f09c730861225d43 2023-04-12 02:54:10,871 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=36 updating hbase:meta row=4e1092f3c00484f8f09c730861225d43, regionState=CLOSED 2023-04-12 02:54:10,871 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testSplitWithoutPONR,,1681268044634.4e1092f3c00484f8f09c730861225d43.","families":{"info":[{"qualifier":"regioninfo","vlen":56,"tag":[],"timestamp":"1681268050871"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681268050871"}]},"ts":"1681268050871"} 2023-04-12 02:54:10,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-12 02:54:10,891 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=39, resume processing ppid=36 2023-04-12 02:54:10,891 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=39, ppid=36, state=SUCCESS; CloseRegionProcedure 4e1092f3c00484f8f09c730861225d43, server=jenkins-hbase4.apache.org,34579,1681268031719 in 270 msec 2023-04-12 02:54:10,899 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=36, resume processing ppid=35 2023-04-12 02:54:10,900 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=36, ppid=35, state=SUCCESS; TransitRegionStateProcedure table=testSplitWithoutPONR, region=4e1092f3c00484f8f09c730861225d43, UNASSIGN in 299 msec 2023-04-12 02:54:10,901 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testSplitWithoutPONR","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681268050901"}]},"ts":"1681268050901"} 2023-04-12 02:54:10,903 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testSplitWithoutPONR, state=DISABLED in hbase:meta 2023-04-12 02:54:10,905 INFO [PEWorker-1] procedure.DisableTableProcedure(305): Set testSplitWithoutPONR to state=DISABLED 2023-04-12 02:54:10,908 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=35, state=SUCCESS; DisableTableProcedure table=testSplitWithoutPONR in 326 msec 2023-04-12 02:54:11,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-12 02:54:11,190 INFO [Listener at localhost/46667] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testSplitWithoutPONR, procId: 35 completed 2023-04-12 02:54:11,191 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete testSplitWithoutPONR 2023-04-12 02:54:11,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] procedure2.ProcedureExecutor(1029): Stored pid=40, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=testSplitWithoutPONR 2023-04-12 02:54:11,195 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=40, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=testSplitWithoutPONR 2023-04-12 02:54:11,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] master.MasterRpcServices(1227): Checking to see if procedure is done pid=40 2023-04-12 02:54:11,196 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=40, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=testSplitWithoutPONR 2023-04-12 02:54:11,203 DEBUG [HFileArchiver-7] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d 2023-04-12 02:54:11,203 DEBUG [HFileArchiver-8] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitWithoutPONR/4e1092f3c00484f8f09c730861225d43 2023-04-12 02:54:11,204 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitWithoutPONR/291734c00d212fd203b04cf8872d4649 2023-04-12 02:54:11,207 DEBUG [HFileArchiver-7] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf1, FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf2, FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/recovered.edits] 2023-04-12 02:54:11,208 DEBUG [HFileArchiver-1] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitWithoutPONR/291734c00d212fd203b04cf8872d4649/291734c00d212fd203b04cf8872d4649, FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitWithoutPONR/291734c00d212fd203b04cf8872d4649/cf1, FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitWithoutPONR/291734c00d212fd203b04cf8872d4649/cf2, FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitWithoutPONR/291734c00d212fd203b04cf8872d4649/recovered.edits] 2023-04-12 02:54:11,208 DEBUG [HFileArchiver-8] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitWithoutPONR/4e1092f3c00484f8f09c730861225d43/4e1092f3c00484f8f09c730861225d43, FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitWithoutPONR/4e1092f3c00484f8f09c730861225d43/cf1, FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitWithoutPONR/4e1092f3c00484f8f09c730861225d43/cf2, FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitWithoutPONR/4e1092f3c00484f8f09c730861225d43/recovered.edits] 2023-04-12 02:54:11,222 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitWithoutPONR/291734c00d212fd203b04cf8872d4649/291734c00d212fd203b04cf8872d4649/.regioninfo to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitWithoutPONR/291734c00d212fd203b04cf8872d4649/291734c00d212fd203b04cf8872d4649/.regioninfo 2023-04-12 02:54:11,222 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf1/0deaf4065ec441859d88faeb52cfa266 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf1/0deaf4065ec441859d88faeb52cfa266 2023-04-12 02:54:11,232 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitWithoutPONR/4e1092f3c00484f8f09c730861225d43/4e1092f3c00484f8f09c730861225d43/.regioninfo to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitWithoutPONR/4e1092f3c00484f8f09c730861225d43/4e1092f3c00484f8f09c730861225d43/.regioninfo 2023-04-12 02:54:11,235 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf1/1cabdcde78c14f6a8ec70790c1ae4cc4 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf1/1cabdcde78c14f6a8ec70790c1ae4cc4 2023-04-12 02:54:11,237 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitWithoutPONR/291734c00d212fd203b04cf8872d4649/291734c00d212fd203b04cf8872d4649/cf1/0deaf4065ec441859d88faeb52cfa266.315e13e868a7f68a3ce37802b2f6312d to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitWithoutPONR/291734c00d212fd203b04cf8872d4649/291734c00d212fd203b04cf8872d4649/cf1/0deaf4065ec441859d88faeb52cfa266.315e13e868a7f68a3ce37802b2f6312d 2023-04-12 02:54:11,238 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf1/2ca08ea408834a75bc73ef0d3d78fa73 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf1/2ca08ea408834a75bc73ef0d3d78fa73 2023-04-12 02:54:11,240 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitWithoutPONR/291734c00d212fd203b04cf8872d4649/291734c00d212fd203b04cf8872d4649/cf1/1cabdcde78c14f6a8ec70790c1ae4cc4.315e13e868a7f68a3ce37802b2f6312d to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitWithoutPONR/291734c00d212fd203b04cf8872d4649/291734c00d212fd203b04cf8872d4649/cf1/1cabdcde78c14f6a8ec70790c1ae4cc4.315e13e868a7f68a3ce37802b2f6312d 2023-04-12 02:54:11,241 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitWithoutPONR/4e1092f3c00484f8f09c730861225d43/4e1092f3c00484f8f09c730861225d43/cf1/0deaf4065ec441859d88faeb52cfa266.315e13e868a7f68a3ce37802b2f6312d to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitWithoutPONR/4e1092f3c00484f8f09c730861225d43/4e1092f3c00484f8f09c730861225d43/cf1/0deaf4065ec441859d88faeb52cfa266.315e13e868a7f68a3ce37802b2f6312d 2023-04-12 02:54:11,241 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf1/2fd181776c3749c18358aebdfc597014 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf1/2fd181776c3749c18358aebdfc597014 2023-04-12 02:54:11,242 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitWithoutPONR/291734c00d212fd203b04cf8872d4649/291734c00d212fd203b04cf8872d4649/cf1/2ca08ea408834a75bc73ef0d3d78fa73.315e13e868a7f68a3ce37802b2f6312d to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitWithoutPONR/291734c00d212fd203b04cf8872d4649/291734c00d212fd203b04cf8872d4649/cf1/2ca08ea408834a75bc73ef0d3d78fa73.315e13e868a7f68a3ce37802b2f6312d 2023-04-12 02:54:11,243 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitWithoutPONR/4e1092f3c00484f8f09c730861225d43/4e1092f3c00484f8f09c730861225d43/cf1/1cabdcde78c14f6a8ec70790c1ae4cc4.315e13e868a7f68a3ce37802b2f6312d to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitWithoutPONR/4e1092f3c00484f8f09c730861225d43/4e1092f3c00484f8f09c730861225d43/cf1/1cabdcde78c14f6a8ec70790c1ae4cc4.315e13e868a7f68a3ce37802b2f6312d 2023-04-12 02:54:11,245 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitWithoutPONR/291734c00d212fd203b04cf8872d4649/291734c00d212fd203b04cf8872d4649/cf1/2fd181776c3749c18358aebdfc597014.315e13e868a7f68a3ce37802b2f6312d to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitWithoutPONR/291734c00d212fd203b04cf8872d4649/291734c00d212fd203b04cf8872d4649/cf1/2fd181776c3749c18358aebdfc597014.315e13e868a7f68a3ce37802b2f6312d 2023-04-12 02:54:11,246 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf1/55bdd10dae634a008e897b8e86048f3c to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf1/55bdd10dae634a008e897b8e86048f3c 2023-04-12 02:54:11,247 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitWithoutPONR/4e1092f3c00484f8f09c730861225d43/4e1092f3c00484f8f09c730861225d43/cf1/2ca08ea408834a75bc73ef0d3d78fa73.315e13e868a7f68a3ce37802b2f6312d to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitWithoutPONR/4e1092f3c00484f8f09c730861225d43/4e1092f3c00484f8f09c730861225d43/cf1/2ca08ea408834a75bc73ef0d3d78fa73.315e13e868a7f68a3ce37802b2f6312d 2023-04-12 02:54:11,248 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitWithoutPONR/291734c00d212fd203b04cf8872d4649/291734c00d212fd203b04cf8872d4649/cf1/55bdd10dae634a008e897b8e86048f3c.315e13e868a7f68a3ce37802b2f6312d to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitWithoutPONR/291734c00d212fd203b04cf8872d4649/291734c00d212fd203b04cf8872d4649/cf1/55bdd10dae634a008e897b8e86048f3c.315e13e868a7f68a3ce37802b2f6312d 2023-04-12 02:54:11,249 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf1/713becfd985e40a8892fbc1f6fe6bdb4 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf1/713becfd985e40a8892fbc1f6fe6bdb4 2023-04-12 02:54:11,251 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitWithoutPONR/291734c00d212fd203b04cf8872d4649/291734c00d212fd203b04cf8872d4649/cf1/713becfd985e40a8892fbc1f6fe6bdb4.315e13e868a7f68a3ce37802b2f6312d to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitWithoutPONR/291734c00d212fd203b04cf8872d4649/291734c00d212fd203b04cf8872d4649/cf1/713becfd985e40a8892fbc1f6fe6bdb4.315e13e868a7f68a3ce37802b2f6312d 2023-04-12 02:54:11,252 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitWithoutPONR/4e1092f3c00484f8f09c730861225d43/4e1092f3c00484f8f09c730861225d43/cf1/2fd181776c3749c18358aebdfc597014.315e13e868a7f68a3ce37802b2f6312d to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitWithoutPONR/4e1092f3c00484f8f09c730861225d43/4e1092f3c00484f8f09c730861225d43/cf1/2fd181776c3749c18358aebdfc597014.315e13e868a7f68a3ce37802b2f6312d 2023-04-12 02:54:11,252 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf1/7e0f8c9c61e247ecbfc4c27206fae340 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf1/7e0f8c9c61e247ecbfc4c27206fae340 2023-04-12 02:54:11,254 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitWithoutPONR/291734c00d212fd203b04cf8872d4649/291734c00d212fd203b04cf8872d4649/cf1/7e0f8c9c61e247ecbfc4c27206fae340.315e13e868a7f68a3ce37802b2f6312d to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitWithoutPONR/291734c00d212fd203b04cf8872d4649/291734c00d212fd203b04cf8872d4649/cf1/7e0f8c9c61e247ecbfc4c27206fae340.315e13e868a7f68a3ce37802b2f6312d 2023-04-12 02:54:11,255 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitWithoutPONR/4e1092f3c00484f8f09c730861225d43/4e1092f3c00484f8f09c730861225d43/cf1/55bdd10dae634a008e897b8e86048f3c.315e13e868a7f68a3ce37802b2f6312d to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitWithoutPONR/4e1092f3c00484f8f09c730861225d43/4e1092f3c00484f8f09c730861225d43/cf1/55bdd10dae634a008e897b8e86048f3c.315e13e868a7f68a3ce37802b2f6312d 2023-04-12 02:54:11,257 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitWithoutPONR/4e1092f3c00484f8f09c730861225d43/4e1092f3c00484f8f09c730861225d43/cf1/713becfd985e40a8892fbc1f6fe6bdb4.315e13e868a7f68a3ce37802b2f6312d to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitWithoutPONR/4e1092f3c00484f8f09c730861225d43/4e1092f3c00484f8f09c730861225d43/cf1/713becfd985e40a8892fbc1f6fe6bdb4.315e13e868a7f68a3ce37802b2f6312d 2023-04-12 02:54:11,260 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitWithoutPONR/4e1092f3c00484f8f09c730861225d43/4e1092f3c00484f8f09c730861225d43/cf1/7e0f8c9c61e247ecbfc4c27206fae340.315e13e868a7f68a3ce37802b2f6312d to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitWithoutPONR/4e1092f3c00484f8f09c730861225d43/4e1092f3c00484f8f09c730861225d43/cf1/7e0f8c9c61e247ecbfc4c27206fae340.315e13e868a7f68a3ce37802b2f6312d 2023-04-12 02:54:11,260 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitWithoutPONR/291734c00d212fd203b04cf8872d4649/291734c00d212fd203b04cf8872d4649/cf2/0b28a4ef23e442f38cfdf4760e6f9480.315e13e868a7f68a3ce37802b2f6312d to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitWithoutPONR/291734c00d212fd203b04cf8872d4649/291734c00d212fd203b04cf8872d4649/cf2/0b28a4ef23e442f38cfdf4760e6f9480.315e13e868a7f68a3ce37802b2f6312d 2023-04-12 02:54:11,260 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf2/0b28a4ef23e442f38cfdf4760e6f9480 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf2/0b28a4ef23e442f38cfdf4760e6f9480 2023-04-12 02:54:11,262 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitWithoutPONR/291734c00d212fd203b04cf8872d4649/291734c00d212fd203b04cf8872d4649/cf2/2540a68ffbdf4440b5ff30ca6a36fada.315e13e868a7f68a3ce37802b2f6312d to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitWithoutPONR/291734c00d212fd203b04cf8872d4649/291734c00d212fd203b04cf8872d4649/cf2/2540a68ffbdf4440b5ff30ca6a36fada.315e13e868a7f68a3ce37802b2f6312d 2023-04-12 02:54:11,263 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf2/2540a68ffbdf4440b5ff30ca6a36fada to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf2/2540a68ffbdf4440b5ff30ca6a36fada 2023-04-12 02:54:11,264 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitWithoutPONR/291734c00d212fd203b04cf8872d4649/291734c00d212fd203b04cf8872d4649/cf2/3a80c419c6b94682b44cf75bc8aef7bd.315e13e868a7f68a3ce37802b2f6312d to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitWithoutPONR/291734c00d212fd203b04cf8872d4649/291734c00d212fd203b04cf8872d4649/cf2/3a80c419c6b94682b44cf75bc8aef7bd.315e13e868a7f68a3ce37802b2f6312d 2023-04-12 02:54:11,265 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf2/3a80c419c6b94682b44cf75bc8aef7bd to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf2/3a80c419c6b94682b44cf75bc8aef7bd 2023-04-12 02:54:11,265 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitWithoutPONR/4e1092f3c00484f8f09c730861225d43/4e1092f3c00484f8f09c730861225d43/cf2/0b28a4ef23e442f38cfdf4760e6f9480.315e13e868a7f68a3ce37802b2f6312d to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitWithoutPONR/4e1092f3c00484f8f09c730861225d43/4e1092f3c00484f8f09c730861225d43/cf2/0b28a4ef23e442f38cfdf4760e6f9480.315e13e868a7f68a3ce37802b2f6312d 2023-04-12 02:54:11,268 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitWithoutPONR/4e1092f3c00484f8f09c730861225d43/4e1092f3c00484f8f09c730861225d43/cf2/2540a68ffbdf4440b5ff30ca6a36fada.315e13e868a7f68a3ce37802b2f6312d to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitWithoutPONR/4e1092f3c00484f8f09c730861225d43/4e1092f3c00484f8f09c730861225d43/cf2/2540a68ffbdf4440b5ff30ca6a36fada.315e13e868a7f68a3ce37802b2f6312d 2023-04-12 02:54:11,268 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf2/babf83b74cec452290ff19cd45eb4713 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf2/babf83b74cec452290ff19cd45eb4713 2023-04-12 02:54:11,268 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitWithoutPONR/291734c00d212fd203b04cf8872d4649/291734c00d212fd203b04cf8872d4649/cf2/babf83b74cec452290ff19cd45eb4713.315e13e868a7f68a3ce37802b2f6312d to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitWithoutPONR/291734c00d212fd203b04cf8872d4649/291734c00d212fd203b04cf8872d4649/cf2/babf83b74cec452290ff19cd45eb4713.315e13e868a7f68a3ce37802b2f6312d 2023-04-12 02:54:11,270 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf2/d253c9057ec44edbad8b758da6e18d8e to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf2/d253c9057ec44edbad8b758da6e18d8e 2023-04-12 02:54:11,270 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitWithoutPONR/4e1092f3c00484f8f09c730861225d43/4e1092f3c00484f8f09c730861225d43/cf2/3a80c419c6b94682b44cf75bc8aef7bd.315e13e868a7f68a3ce37802b2f6312d to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitWithoutPONR/4e1092f3c00484f8f09c730861225d43/4e1092f3c00484f8f09c730861225d43/cf2/3a80c419c6b94682b44cf75bc8aef7bd.315e13e868a7f68a3ce37802b2f6312d 2023-04-12 02:54:11,270 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitWithoutPONR/291734c00d212fd203b04cf8872d4649/291734c00d212fd203b04cf8872d4649/cf2/d253c9057ec44edbad8b758da6e18d8e.315e13e868a7f68a3ce37802b2f6312d to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitWithoutPONR/291734c00d212fd203b04cf8872d4649/291734c00d212fd203b04cf8872d4649/cf2/d253c9057ec44edbad8b758da6e18d8e.315e13e868a7f68a3ce37802b2f6312d 2023-04-12 02:54:11,273 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf2/d5ab6ea45d1b4298ae7d38df0de5a43f to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf2/d5ab6ea45d1b4298ae7d38df0de5a43f 2023-04-12 02:54:11,273 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitWithoutPONR/291734c00d212fd203b04cf8872d4649/291734c00d212fd203b04cf8872d4649/cf2/d5ab6ea45d1b4298ae7d38df0de5a43f.315e13e868a7f68a3ce37802b2f6312d to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitWithoutPONR/291734c00d212fd203b04cf8872d4649/291734c00d212fd203b04cf8872d4649/cf2/d5ab6ea45d1b4298ae7d38df0de5a43f.315e13e868a7f68a3ce37802b2f6312d 2023-04-12 02:54:11,274 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitWithoutPONR/4e1092f3c00484f8f09c730861225d43/4e1092f3c00484f8f09c730861225d43/cf2/babf83b74cec452290ff19cd45eb4713.315e13e868a7f68a3ce37802b2f6312d to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitWithoutPONR/4e1092f3c00484f8f09c730861225d43/4e1092f3c00484f8f09c730861225d43/cf2/babf83b74cec452290ff19cd45eb4713.315e13e868a7f68a3ce37802b2f6312d 2023-04-12 02:54:11,275 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf2/f13ea6a8012d44709becae664369d2d0 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/cf2/f13ea6a8012d44709becae664369d2d0 2023-04-12 02:54:11,276 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitWithoutPONR/291734c00d212fd203b04cf8872d4649/291734c00d212fd203b04cf8872d4649/cf2/f13ea6a8012d44709becae664369d2d0.315e13e868a7f68a3ce37802b2f6312d to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitWithoutPONR/291734c00d212fd203b04cf8872d4649/291734c00d212fd203b04cf8872d4649/cf2/f13ea6a8012d44709becae664369d2d0.315e13e868a7f68a3ce37802b2f6312d 2023-04-12 02:54:11,280 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitWithoutPONR/291734c00d212fd203b04cf8872d4649/cf1/7a1a6c7dd03543e880b73d3a48fa09bc to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitWithoutPONR/291734c00d212fd203b04cf8872d4649/cf1/7a1a6c7dd03543e880b73d3a48fa09bc 2023-04-12 02:54:11,281 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitWithoutPONR/4e1092f3c00484f8f09c730861225d43/4e1092f3c00484f8f09c730861225d43/cf2/d253c9057ec44edbad8b758da6e18d8e.315e13e868a7f68a3ce37802b2f6312d to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitWithoutPONR/4e1092f3c00484f8f09c730861225d43/4e1092f3c00484f8f09c730861225d43/cf2/d253c9057ec44edbad8b758da6e18d8e.315e13e868a7f68a3ce37802b2f6312d 2023-04-12 02:54:11,281 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/recovered.edits/85.seqid to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d/recovered.edits/85.seqid 2023-04-12 02:54:11,284 DEBUG [HFileArchiver-7] backup.HFileArchiver(596): Deleted hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitWithoutPONR/315e13e868a7f68a3ce37802b2f6312d 2023-04-12 02:54:11,285 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitWithoutPONR/4e1092f3c00484f8f09c730861225d43/4e1092f3c00484f8f09c730861225d43/cf2/d5ab6ea45d1b4298ae7d38df0de5a43f.315e13e868a7f68a3ce37802b2f6312d to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitWithoutPONR/4e1092f3c00484f8f09c730861225d43/4e1092f3c00484f8f09c730861225d43/cf2/d5ab6ea45d1b4298ae7d38df0de5a43f.315e13e868a7f68a3ce37802b2f6312d 2023-04-12 02:54:11,286 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitWithoutPONR/291734c00d212fd203b04cf8872d4649/cf2/8c55c3693a994e108164d71a69ad36d6 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitWithoutPONR/291734c00d212fd203b04cf8872d4649/cf2/8c55c3693a994e108164d71a69ad36d6 2023-04-12 02:54:11,286 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitWithoutPONR/4e1092f3c00484f8f09c730861225d43/4e1092f3c00484f8f09c730861225d43/cf2/f13ea6a8012d44709becae664369d2d0.315e13e868a7f68a3ce37802b2f6312d to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitWithoutPONR/4e1092f3c00484f8f09c730861225d43/4e1092f3c00484f8f09c730861225d43/cf2/f13ea6a8012d44709becae664369d2d0.315e13e868a7f68a3ce37802b2f6312d 2023-04-12 02:54:11,290 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitWithoutPONR/291734c00d212fd203b04cf8872d4649/recovered.edits/90.seqid to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitWithoutPONR/291734c00d212fd203b04cf8872d4649/recovered.edits/90.seqid 2023-04-12 02:54:11,292 DEBUG [HFileArchiver-1] backup.HFileArchiver(596): Deleted hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitWithoutPONR/291734c00d212fd203b04cf8872d4649 2023-04-12 02:54:11,292 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitWithoutPONR/4e1092f3c00484f8f09c730861225d43/cf1/fcf8238b62664b00a68f527d6561a646 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitWithoutPONR/4e1092f3c00484f8f09c730861225d43/cf1/fcf8238b62664b00a68f527d6561a646 2023-04-12 02:54:11,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] master.MasterRpcServices(1227): Checking to see if procedure is done pid=40 2023-04-12 02:54:11,297 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitWithoutPONR/4e1092f3c00484f8f09c730861225d43/cf2/63fd42795d264c699b7793e9ff2e314f to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitWithoutPONR/4e1092f3c00484f8f09c730861225d43/cf2/63fd42795d264c699b7793e9ff2e314f 2023-04-12 02:54:11,301 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitWithoutPONR/4e1092f3c00484f8f09c730861225d43/recovered.edits/90.seqid to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitWithoutPONR/4e1092f3c00484f8f09c730861225d43/recovered.edits/90.seqid 2023-04-12 02:54:11,301 DEBUG [HFileArchiver-8] backup.HFileArchiver(596): Deleted hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitWithoutPONR/4e1092f3c00484f8f09c730861225d43 2023-04-12 02:54:11,301 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived testSplitWithoutPONR regions 2023-04-12 02:54:11,304 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=40, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=testSplitWithoutPONR 2023-04-12 02:54:11,307 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 3 rows of testSplitWithoutPONR from hbase:meta 2023-04-12 02:54:11,310 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'testSplitWithoutPONR' descriptor. 2023-04-12 02:54:11,312 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=40, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=testSplitWithoutPONR 2023-04-12 02:54:11,312 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'testSplitWithoutPONR' from region states. 2023-04-12 02:54:11,312 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testSplitWithoutPONR,,1681268041279.315e13e868a7f68a3ce37802b2f6312d.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1681268051312"}]},"ts":"9223372036854775807"} 2023-04-12 02:54:11,312 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testSplitWithoutPONR,,1681268044634.4e1092f3c00484f8f09c730861225d43.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1681268051312"}]},"ts":"9223372036854775807"} 2023-04-12 02:54:11,312 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testSplitWithoutPONR,41,1681268044634.291734c00d212fd203b04cf8872d4649.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1681268051312"}]},"ts":"9223372036854775807"} 2023-04-12 02:54:11,316 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 3 regions from META 2023-04-12 02:54:11,316 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 315e13e868a7f68a3ce37802b2f6312d, NAME => 'testSplitWithoutPONR,,1681268041279.315e13e868a7f68a3ce37802b2f6312d.', STARTKEY => '', ENDKEY => ''}, {ENCODED => 4e1092f3c00484f8f09c730861225d43, NAME => 'testSplitWithoutPONR,,1681268044634.4e1092f3c00484f8f09c730861225d43.', STARTKEY => '', ENDKEY => '41'}, {ENCODED => 291734c00d212fd203b04cf8872d4649, NAME => 'testSplitWithoutPONR,41,1681268044634.291734c00d212fd203b04cf8872d4649.', STARTKEY => '41', ENDKEY => ''}] 2023-04-12 02:54:11,316 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'testSplitWithoutPONR' as deleted. 2023-04-12 02:54:11,316 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testSplitWithoutPONR","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1681268051316"}]},"ts":"9223372036854775807"} 2023-04-12 02:54:11,318 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table testSplitWithoutPONR state from META 2023-04-12 02:54:11,320 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=40, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=testSplitWithoutPONR 2023-04-12 02:54:11,321 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=40, state=SUCCESS; DeleteTableProcedure table=testSplitWithoutPONR in 129 msec 2023-04-12 02:54:11,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] master.MasterRpcServices(1227): Checking to see if procedure is done pid=40 2023-04-12 02:54:11,498 INFO [Listener at localhost/46667] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:testSplitWithoutPONR, procId: 40 completed 2023-04-12 02:54:11,519 INFO [Listener at localhost/46667] hbase.ResourceChecker(175): after: master.assignment.TestSplitTableRegionProcedure#testSplitWithoutPONR Thread=402 (was 390) Potentially hanging thread: RS:0;jenkins-hbase4:34579-shortCompactions-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.PriorityBlockingQueue.take(PriorityBlockingQueue.java:549) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/22e25865-4487-e2aa-7556-0db30ee3c8ff/cluster_2d9c59b9-d025-040c-6051-62d17e49a4b2/dfs/data/data4/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x6cb27db9-shared-pool-14 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1018420944_15 at /127.0.0.1:56346 [Waiting for operation #5] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1018420944_15 at /127.0.0.1:46744 [Waiting for operation #29] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1018420944_15 at /127.0.0.1:40224 [Waiting for operation #19] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1018420944_15 at /127.0.0.1:46690 [Waiting for operation #32] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/22e25865-4487-e2aa-7556-0db30ee3c8ff/cluster_2d9c59b9-d025-040c-6051-62d17e49a4b2/dfs/data/data2/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x2d8012cd-shared-pool-6 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1018420944_15 at /127.0.0.1:37110 [Waiting for operation #11] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1018420944_15 at /127.0.0.1:56332 [Waiting for operation #6] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x6cb27db9-shared-pool-13 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x6cb27db9-shared-pool-15 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x6cb27db9-shared-pool-18 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-3 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:883) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/22e25865-4487-e2aa-7556-0db30ee3c8ff/cluster_2d9c59b9-d025-040c-6051-62d17e49a4b2/dfs/data/data3/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-3-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1018420944_15 at /127.0.0.1:36588 [Waiting for operation #24] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1018420944_15 at /127.0.0.1:37064 [Waiting for operation #15] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x6cb27db9-shared-pool-16 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-6 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1018420944_15 at /127.0.0.1:40818 [Waiting for operation #2] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/22e25865-4487-e2aa-7556-0db30ee3c8ff/cluster_2d9c59b9-d025-040c-6051-62d17e49a4b2/dfs/data/data1/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1018420944_15 at /127.0.0.1:41308 [Waiting for operation #5] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x2d8012cd-metaLookup-shared--pool-5 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1018420944_15 at /127.0.0.1:37042 [Waiting for operation #11] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1018420944_15 at /127.0.0.1:37084 [Waiting for operation #18] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/22e25865-4487-e2aa-7556-0db30ee3c8ff/cluster_2d9c59b9-d025-040c-6051-62d17e49a4b2/dfs/data/data5/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-8 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-3-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x2d8012cd-shared-pool-7 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1018420944_15 at /127.0.0.1:40252 [Waiting for operation #19] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1018420944_15 at /127.0.0.1:56356 [Waiting for operation #6] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/22e25865-4487-e2aa-7556-0db30ee3c8ff/cluster_2d9c59b9-d025-040c-6051-62d17e49a4b2/dfs/data/data6/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1018420944_15 at /127.0.0.1:46616 [Waiting for operation #32] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x6cb27db9-shared-pool-17 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x2d8012cd-shared-pool-5 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1018420944_15 at /127.0.0.1:40180 [Waiting for operation #19] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x2d8012cd-shared-pool-8 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-7 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x2d8012cd-metaLookup-shared--pool-4 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) - Thread LEAK? -, OpenFileDescriptor=826 (was 844), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=457 (was 376) - SystemLoadAverage LEAK? -, ProcessCount=172 (was 169) - ProcessCount LEAK? -, AvailableMemoryMB=1225 (was 1477) 2023-04-12 02:54:11,541 INFO [Listener at localhost/46667] hbase.ResourceChecker(147): before: master.assignment.TestSplitTableRegionProcedure#testRollbackAndDoubleExecution Thread=402, OpenFileDescriptor=826, MaxFileDescriptor=60000, SystemLoadAverage=457, ProcessCount=172, AvailableMemoryMB=1224 2023-04-12 02:54:11,542 WARN [Listener at localhost/46667] procedure2.ProcedureTestingUtility(193): Set Kill before store update to: false 2023-04-12 02:54:11,551 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/balancer 2023-04-12 02:54:11,552 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] master.MasterRpcServices(492): Client=jenkins//172.31.14.131 set balanceSwitch=false 2023-04-12 02:54:11,554 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(1029): Stored pid=41, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=testRollbackAndDoubleExecution 2023-04-12 02:54:11,557 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=41, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=testRollbackAndDoubleExecution execute state=CREATE_TABLE_PRE_OPERATION 2023-04-12 02:54:11,558 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=41, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=testRollbackAndDoubleExecution execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-12 02:54:11,560 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a 2023-04-12 02:54:11,561 DEBUG [HFileArchiver-2] backup.HFileArchiver(153): Directory hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a empty. 2023-04-12 02:54:11,561 DEBUG [HFileArchiver-2] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a 2023-04-12 02:54:11,561 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived testRollbackAndDoubleExecution regions 2023-04-12 02:54:11,592 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-12 02:54:11,638 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testRollbackAndDoubleExecution/.tabledesc/.tableinfo.0000000001 2023-04-12 02:54:11,645 INFO [RegionOpenAndInit-testRollbackAndDoubleExecution-pool-0] regionserver.HRegion(7675): creating {ENCODED => 96cdae2f7638212730b694522b9a464a, NAME => 'testRollbackAndDoubleExecution,,1681268051553.96cdae2f7638212730b694522b9a464a.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='testRollbackAndDoubleExecution', {NAME => 'cf1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'cf2', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp 2023-04-12 02:54:11,674 DEBUG [RegionOpenAndInit-testRollbackAndDoubleExecution-pool-0] regionserver.HRegion(866): Instantiated testRollbackAndDoubleExecution,,1681268051553.96cdae2f7638212730b694522b9a464a.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-12 02:54:11,674 DEBUG [RegionOpenAndInit-testRollbackAndDoubleExecution-pool-0] regionserver.HRegion(1604): Closing 96cdae2f7638212730b694522b9a464a, disabling compactions & flushes 2023-04-12 02:54:11,674 INFO [RegionOpenAndInit-testRollbackAndDoubleExecution-pool-0] regionserver.HRegion(1626): Closing region testRollbackAndDoubleExecution,,1681268051553.96cdae2f7638212730b694522b9a464a. 2023-04-12 02:54:11,674 DEBUG [RegionOpenAndInit-testRollbackAndDoubleExecution-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testRollbackAndDoubleExecution,,1681268051553.96cdae2f7638212730b694522b9a464a. 2023-04-12 02:54:11,674 DEBUG [RegionOpenAndInit-testRollbackAndDoubleExecution-pool-0] regionserver.HRegion(1714): Acquired close lock on testRollbackAndDoubleExecution,,1681268051553.96cdae2f7638212730b694522b9a464a. after waiting 0 ms 2023-04-12 02:54:11,674 DEBUG [RegionOpenAndInit-testRollbackAndDoubleExecution-pool-0] regionserver.HRegion(1724): Updates disabled for region testRollbackAndDoubleExecution,,1681268051553.96cdae2f7638212730b694522b9a464a. 2023-04-12 02:54:11,674 INFO [RegionOpenAndInit-testRollbackAndDoubleExecution-pool-0] regionserver.HRegion(1838): Closed testRollbackAndDoubleExecution,,1681268051553.96cdae2f7638212730b694522b9a464a. 2023-04-12 02:54:11,674 DEBUG [RegionOpenAndInit-testRollbackAndDoubleExecution-pool-0] regionserver.HRegion(1558): Region close journal for 96cdae2f7638212730b694522b9a464a: 2023-04-12 02:54:11,678 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=41, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=testRollbackAndDoubleExecution execute state=CREATE_TABLE_ADD_TO_META 2023-04-12 02:54:11,679 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRollbackAndDoubleExecution,,1681268051553.96cdae2f7638212730b694522b9a464a.","families":{"info":[{"qualifier":"regioninfo","vlen":64,"tag":[],"timestamp":"1681268051679"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681268051679"}]},"ts":"1681268051679"} 2023-04-12 02:54:11,682 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-12 02:54:11,683 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=41, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=testRollbackAndDoubleExecution execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-12 02:54:11,683 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRollbackAndDoubleExecution","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681268051683"}]},"ts":"1681268051683"} 2023-04-12 02:54:11,685 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testRollbackAndDoubleExecution, state=ENABLING in hbase:meta 2023-04-12 02:54:11,689 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-04-12 02:54:11,691 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-04-12 02:54:11,691 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-04-12 02:54:11,691 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-04-12 02:54:11,691 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-04-12 02:54:11,691 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=42, ppid=41, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=96cdae2f7638212730b694522b9a464a, ASSIGN}] 2023-04-12 02:54:11,693 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=42, ppid=41, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=96cdae2f7638212730b694522b9a464a, ASSIGN 2023-04-12 02:54:11,694 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=42, ppid=41, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=96cdae2f7638212730b694522b9a464a, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,44679,1681268031772; forceNewPlan=false, retain=false 2023-04-12 02:54:11,844 INFO [jenkins-hbase4:40281] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-04-12 02:54:11,846 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=42 updating hbase:meta row=96cdae2f7638212730b694522b9a464a, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:11,846 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRollbackAndDoubleExecution,,1681268051553.96cdae2f7638212730b694522b9a464a.","families":{"info":[{"qualifier":"regioninfo","vlen":64,"tag":[],"timestamp":"1681268051845"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681268051845"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681268051845"}]},"ts":"1681268051845"} 2023-04-12 02:54:11,848 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=43, ppid=42, state=RUNNABLE; OpenRegionProcedure 96cdae2f7638212730b694522b9a464a, server=jenkins-hbase4.apache.org,44679,1681268031772}] 2023-04-12 02:54:12,008 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRollbackAndDoubleExecution,,1681268051553.96cdae2f7638212730b694522b9a464a. 2023-04-12 02:54:12,008 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 96cdae2f7638212730b694522b9a464a, NAME => 'testRollbackAndDoubleExecution,,1681268051553.96cdae2f7638212730b694522b9a464a.', STARTKEY => '', ENDKEY => ''} 2023-04-12 02:54:12,009 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(174): System coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro loaded, priority=536870911. 2023-04-12 02:54:12,009 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRollbackAndDoubleExecution 96cdae2f7638212730b694522b9a464a 2023-04-12 02:54:12,009 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated testRollbackAndDoubleExecution,,1681268051553.96cdae2f7638212730b694522b9a464a.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-12 02:54:12,009 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 96cdae2f7638212730b694522b9a464a 2023-04-12 02:54:12,009 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 96cdae2f7638212730b694522b9a464a 2023-04-12 02:54:12,011 INFO [StoreOpener-96cdae2f7638212730b694522b9a464a-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf1 of region 96cdae2f7638212730b694522b9a464a 2023-04-12 02:54:12,012 DEBUG [StoreOpener-96cdae2f7638212730b694522b9a464a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/cf1 2023-04-12 02:54:12,012 DEBUG [StoreOpener-96cdae2f7638212730b694522b9a464a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/cf1 2023-04-12 02:54:12,013 INFO [StoreOpener-96cdae2f7638212730b694522b9a464a-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 96cdae2f7638212730b694522b9a464a columnFamilyName cf1 2023-04-12 02:54:12,013 INFO [StoreOpener-96cdae2f7638212730b694522b9a464a-1] regionserver.HStore(310): Store=96cdae2f7638212730b694522b9a464a/cf1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-12 02:54:12,013 INFO [StoreOpener-96cdae2f7638212730b694522b9a464a-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf2 of region 96cdae2f7638212730b694522b9a464a 2023-04-12 02:54:12,015 DEBUG [StoreOpener-96cdae2f7638212730b694522b9a464a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/cf2 2023-04-12 02:54:12,015 DEBUG [StoreOpener-96cdae2f7638212730b694522b9a464a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/cf2 2023-04-12 02:54:12,015 INFO [StoreOpener-96cdae2f7638212730b694522b9a464a-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 96cdae2f7638212730b694522b9a464a columnFamilyName cf2 2023-04-12 02:54:12,016 INFO [StoreOpener-96cdae2f7638212730b694522b9a464a-1] regionserver.HStore(310): Store=96cdae2f7638212730b694522b9a464a/cf2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-12 02:54:12,017 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a 2023-04-12 02:54:12,017 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a 2023-04-12 02:54:12,020 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testRollbackAndDoubleExecution descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-04-12 02:54:12,021 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 96cdae2f7638212730b694522b9a464a 2023-04-12 02:54:12,023 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-12 02:54:12,024 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 96cdae2f7638212730b694522b9a464a; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=63995012, jitterRate=-0.04640001058578491}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-04-12 02:54:12,024 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 96cdae2f7638212730b694522b9a464a: 2023-04-12 02:54:12,025 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRollbackAndDoubleExecution,,1681268051553.96cdae2f7638212730b694522b9a464a., pid=43, masterSystemTime=1681268052000 2023-04-12 02:54:12,026 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRollbackAndDoubleExecution,,1681268051553.96cdae2f7638212730b694522b9a464a. 2023-04-12 02:54:12,026 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRollbackAndDoubleExecution,,1681268051553.96cdae2f7638212730b694522b9a464a. 2023-04-12 02:54:12,027 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=42 updating hbase:meta row=96cdae2f7638212730b694522b9a464a, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:12,027 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRollbackAndDoubleExecution,,1681268051553.96cdae2f7638212730b694522b9a464a.","families":{"info":[{"qualifier":"regioninfo","vlen":64,"tag":[],"timestamp":"1681268052027"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1681268052027"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681268052027"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681268052027"}]},"ts":"1681268052027"} 2023-04-12 02:54:12,031 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=43, resume processing ppid=42 2023-04-12 02:54:12,031 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=43, ppid=42, state=SUCCESS; OpenRegionProcedure 96cdae2f7638212730b694522b9a464a, server=jenkins-hbase4.apache.org,44679,1681268031772 in 181 msec 2023-04-12 02:54:12,033 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=42, resume processing ppid=41 2023-04-12 02:54:12,033 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=42, ppid=41, state=SUCCESS; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=96cdae2f7638212730b694522b9a464a, ASSIGN in 340 msec 2023-04-12 02:54:12,034 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=41, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=testRollbackAndDoubleExecution execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-12 02:54:12,034 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRollbackAndDoubleExecution","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681268052034"}]},"ts":"1681268052034"} 2023-04-12 02:54:12,036 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testRollbackAndDoubleExecution, state=ENABLED in hbase:meta 2023-04-12 02:54:12,047 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=41, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=testRollbackAndDoubleExecution execute state=CREATE_TABLE_POST_OPERATION 2023-04-12 02:54:12,049 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=41, state=SUCCESS; CreateTableProcedure table=testRollbackAndDoubleExecution in 494 msec 2023-04-12 02:54:12,064 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 02:54:12,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] procedure.ProcedureCoordinator(165): Submitting procedure testRollbackAndDoubleExecution 2023-04-12 02:54:12,066 INFO [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testRollbackAndDoubleExecution' 2023-04-12 02:54:12,066 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:54:12,066 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testRollbackAndDoubleExecution' starting 'acquire' 2023-04-12 02:54:12,066 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testRollbackAndDoubleExecution', kicking off acquire phase on members. 2023-04-12 02:54:12,067 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-12 02:54:12,067 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-04-12 02:54:12,068 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:54:12,068 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:12,068 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:54:12,068 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:54:12,068 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:54:12,069 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:54:12,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:54:12,069 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:54:12,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:54:12,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:54:12,069 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:12,069 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 02:54:12,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-04-12 02:54:12,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-04-12 02:54:12,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-04-12 02:54:12,069 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-12 02:54:12,070 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-12 02:54:12,070 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-12 02:54:12,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 02:54:12,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-04-12 02:54:12,070 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testRollbackAndDoubleExecution 2023-04-12 02:54:12,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 02:54:12,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-04-12 02:54:12,070 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testRollbackAndDoubleExecution 2023-04-12 02:54:12,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 02:54:12,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-04-12 02:54:12,070 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testRollbackAndDoubleExecution 2023-04-12 02:54:12,070 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testRollbackAndDoubleExecution 2023-04-12 02:54:12,070 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testRollbackAndDoubleExecution' with timeout 60000ms 2023-04-12 02:54:12,070 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:54:12,070 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testRollbackAndDoubleExecution 2023-04-12 02:54:12,071 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testRollbackAndDoubleExecution 2023-04-12 02:54:12,070 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testRollbackAndDoubleExecution' starting 'acquire' stage 2023-04-12 02:54:12,071 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testRollbackAndDoubleExecution' with timeout 60000ms 2023-04-12 02:54:12,071 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:54:12,071 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testRollbackAndDoubleExecution' with timeout 60000ms 2023-04-12 02:54:12,071 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:54:12,071 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testRollbackAndDoubleExecution' starting 'acquire' stage 2023-04-12 02:54:12,071 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testRollbackAndDoubleExecution' locally acquired 2023-04-12 02:54:12,071 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34579,1681268031719' joining acquired barrier for procedure (testRollbackAndDoubleExecution) in zk 2023-04-12 02:54:12,071 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testRollbackAndDoubleExecution' starting 'acquire' stage 2023-04-12 02:54:12,071 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testRollbackAndDoubleExecution' locally acquired 2023-04-12 02:54:12,071 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43091,1681268031804' joining acquired barrier for procedure (testRollbackAndDoubleExecution) in zk 2023-04-12 02:54:12,071 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 02:54:12,071 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 02:54:12,071 DEBUG [rs(jenkins-hbase4.apache.org,44679,1681268031772)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testRollbackAndDoubleExecution,,1681268051553.96cdae2f7638212730b694522b9a464a. 2023-04-12 02:54:12,071 DEBUG [rs(jenkins-hbase4.apache.org,44679,1681268031772)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testRollbackAndDoubleExecution,,1681268051553.96cdae2f7638212730b694522b9a464a. started... 2023-04-12 02:54:12,072 INFO [rs(jenkins-hbase4.apache.org,44679,1681268031772)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 96cdae2f7638212730b694522b9a464a 2/2 column families, dataSize=120 B heapSize=928 B 2023-04-12 02:54:12,074 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-12 02:54:12,075 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-12 02:54:12,075 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-12 02:54:12,075 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testRollbackAndDoubleExecution' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 02:54:12,075 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-12 02:54:12,075 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testRollbackAndDoubleExecution' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 02:54:12,088 INFO [rs(jenkins-hbase4.apache.org,44679,1681268031772)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=60 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/.tmp/cf1/6fbe235c9d994e648689a95a2833784a 2023-04-12 02:54:12,103 INFO [rs(jenkins-hbase4.apache.org,44679,1681268031772)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=60 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/.tmp/cf2/28ceff5b76a942bd8ce4bf8f9ffcc8c9 2023-04-12 02:54:12,114 DEBUG [rs(jenkins-hbase4.apache.org,44679,1681268031772)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/.tmp/cf1/6fbe235c9d994e648689a95a2833784a as hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/cf1/6fbe235c9d994e648689a95a2833784a 2023-04-12 02:54:12,123 INFO [rs(jenkins-hbase4.apache.org,44679,1681268031772)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/cf1/6fbe235c9d994e648689a95a2833784a, entries=2, sequenceid=6, filesize=4.8 K 2023-04-12 02:54:12,131 DEBUG [rs(jenkins-hbase4.apache.org,44679,1681268031772)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/.tmp/cf2/28ceff5b76a942bd8ce4bf8f9ffcc8c9 as hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/cf2/28ceff5b76a942bd8ce4bf8f9ffcc8c9 2023-04-12 02:54:12,139 INFO [rs(jenkins-hbase4.apache.org,44679,1681268031772)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/cf2/28ceff5b76a942bd8ce4bf8f9ffcc8c9, entries=2, sequenceid=6, filesize=4.8 K 2023-04-12 02:54:12,141 INFO [rs(jenkins-hbase4.apache.org,44679,1681268031772)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~120 B/120, heapSize ~896 B/896, currentSize=0 B/0 for 96cdae2f7638212730b694522b9a464a in 69ms, sequenceid=6, compaction requested=false 2023-04-12 02:54:12,147 DEBUG [rs(jenkins-hbase4.apache.org,44679,1681268031772)-flush-proc-pool-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'testRollbackAndDoubleExecution' 2023-04-12 02:54:12,147 DEBUG [rs(jenkins-hbase4.apache.org,44679,1681268031772)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 96cdae2f7638212730b694522b9a464a: 2023-04-12 02:54:12,148 DEBUG [rs(jenkins-hbase4.apache.org,44679,1681268031772)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testRollbackAndDoubleExecution,,1681268051553.96cdae2f7638212730b694522b9a464a. 2023-04-12 02:54:12,148 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 02:54:12,148 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 02:54:12,148 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:12,148 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testRollbackAndDoubleExecution' locally acquired 2023-04-12 02:54:12,148 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44679,1681268031772' joining acquired barrier for procedure (testRollbackAndDoubleExecution) in zk 2023-04-12 02:54:12,151 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:12,151 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-12 02:54:12,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:12,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 02:54:12,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 02:54:12,151 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-12 02:54:12,151 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testRollbackAndDoubleExecution' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 02:54:12,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 02:54:12,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 02:54:12,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRollbackAndDoubleExecution 2023-04-12 02:54:12,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:12,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:12,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:12,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 02:54:12,154 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44679,1681268031772' joining acquired barrier for procedure 'testRollbackAndDoubleExecution' on coordinator 2023-04-12 02:54:12,154 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4c7db1a4[Count = 0] remaining members to acquire global barrier 2023-04-12 02:54:12,154 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testRollbackAndDoubleExecution' starting 'in-barrier' execution. 2023-04-12 02:54:12,154 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-12 02:54:12,155 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-12 02:54:12,155 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-12 02:54:12,155 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-12 02:54:12,155 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-12 02:54:12,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-12 02:54:12,155 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-12 02:54:12,156 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:12,156 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 02:54:12,156 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testRollbackAndDoubleExecution' received 'reached' from coordinator. 2023-04-12 02:54:12,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-12 02:54:12,156 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testRollbackAndDoubleExecution' locally completed 2023-04-12 02:54:12,156 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testRollbackAndDoubleExecution' received 'reached' from coordinator. 2023-04-12 02:54:12,156 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testRollbackAndDoubleExecution' locally completed 2023-04-12 02:54:12,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-12 02:54:12,156 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testRollbackAndDoubleExecution' completed for member 'jenkins-hbase4.apache.org,34579,1681268031719' in zk 2023-04-12 02:54:12,156 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testRollbackAndDoubleExecution' completed for member 'jenkins-hbase4.apache.org,43091,1681268031804' in zk 2023-04-12 02:54:12,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-12 02:54:12,156 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testRollbackAndDoubleExecution' received 'reached' from coordinator. 2023-04-12 02:54:12,156 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testRollbackAndDoubleExecution' locally completed 2023-04-12 02:54:12,156 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testRollbackAndDoubleExecution' completed for member 'jenkins-hbase4.apache.org,44679,1681268031772' in zk 2023-04-12 02:54:12,158 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testRollbackAndDoubleExecution' has notified controller of completion 2023-04-12 02:54:12,158 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:54:12,158 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testRollbackAndDoubleExecution' completed. 2023-04-12 02:54:12,158 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:12,159 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testRollbackAndDoubleExecution' has notified controller of completion 2023-04-12 02:54:12,159 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:54:12,158 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testRollbackAndDoubleExecution' has notified controller of completion 2023-04-12 02:54:12,159 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testRollbackAndDoubleExecution' completed. 2023-04-12 02:54:12,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:12,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 02:54:12,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 02:54:12,159 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:54:12,159 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testRollbackAndDoubleExecution' completed. 2023-04-12 02:54:12,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 02:54:12,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 02:54:12,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRollbackAndDoubleExecution 2023-04-12 02:54:12,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:12,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:12,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:12,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 02:54:12,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRollbackAndDoubleExecution 2023-04-12 02:54:12,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:12,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:12,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:12,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testRollbackAndDoubleExecution' member 'jenkins-hbase4.apache.org,44679,1681268031772': 2023-04-12 02:54:12,164 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44679,1681268031772' released barrier for procedure'testRollbackAndDoubleExecution', counting down latch. Waiting for 0 more 2023-04-12 02:54:12,164 INFO [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testRollbackAndDoubleExecution' execution completed 2023-04-12 02:54:12,165 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 02:54:12,165 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 02:54:12,165 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testRollbackAndDoubleExecution 2023-04-12 02:54:12,165 INFO [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testRollbackAndDoubleExecutionincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 02:54:12,173 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-12 02:54:12,173 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-12 02:54:12,173 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-12 02:54:12,173 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-12 02:54:12,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-12 02:54:12,173 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-12 02:54:12,173 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:54:12,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-12 02:54:12,173 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-12 02:54:12,173 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:54:12,173 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:54:12,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-12 02:54:12,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 02:54:12,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 02:54:12,173 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-12 02:54:12,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-12 02:54:12,174 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:54:12,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:54:12,174 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:54:12,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:54:12,174 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:54:12,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:54:12,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-12 02:54:12,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-12 02:54:12,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 02:54:12,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-12 02:54:12,175 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:12,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRollbackAndDoubleExecution 2023-04-12 02:54:12,176 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:12,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 02:54:12,176 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:12,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRollbackAndDoubleExecution 2023-04-12 02:54:12,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:12,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:12,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:12,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 02:54:12,179 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:12,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRollbackAndDoubleExecution 2023-04-12 02:54:12,179 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:12,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:12,179 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:12,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:12,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:12,184 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:54:12,184 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:12,184 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:54:12,184 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-04-12 02:54:12,184 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:54:12,184 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:54:12,184 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:54:12,184 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:54:12,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:54:12,184 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:54:12,184 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:12,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:54:12,185 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:12,185 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-04-12 02:54:12,185 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:12,185 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:54:12,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:54:12,185 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:54:12,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:54:12,185 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:54:12,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:54:12,185 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:54:12,185 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-12 02:54:12,186 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:12,186 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:12,186 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:54:12,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:54:12,186 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-12 02:54:12,187 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-12 02:54:12,187 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:54:12,187 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testRollbackAndDoubleExecution' 2023-04-12 02:54:12,187 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 02:54:12,187 DEBUG [Listener at localhost/46667] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testRollbackAndDoubleExecution'' to complete. (max 20000 ms per retry) 2023-04-12 02:54:12,187 DEBUG [Listener at localhost/46667] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 02:54:12,287 DEBUG [Listener at localhost/46667] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 02:54:12,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 02:54:12,314 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 02:54:12,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] procedure.ProcedureCoordinator(143): Procedure testRollbackAndDoubleExecution was in running list but was completed. Accepting new attempt. 2023-04-12 02:54:12,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] procedure.ProcedureCoordinator(165): Submitting procedure testRollbackAndDoubleExecution 2023-04-12 02:54:12,324 INFO [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testRollbackAndDoubleExecution' 2023-04-12 02:54:12,324 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:54:12,324 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testRollbackAndDoubleExecution' starting 'acquire' 2023-04-12 02:54:12,324 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testRollbackAndDoubleExecution', kicking off acquire phase on members. 2023-04-12 02:54:12,328 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-12 02:54:12,328 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-04-12 02:54:12,329 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:54:12,330 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:54:12,330 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:54:12,330 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:54:12,330 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:12,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:54:12,330 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:54:12,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:54:12,330 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:12,330 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 02:54:12,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-04-12 02:54:12,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-04-12 02:54:12,331 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-12 02:54:12,331 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-12 02:54:12,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 02:54:12,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 02:54:12,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-04-12 02:54:12,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-04-12 02:54:12,331 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testRollbackAndDoubleExecution 2023-04-12 02:54:12,331 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testRollbackAndDoubleExecution 2023-04-12 02:54:12,332 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testRollbackAndDoubleExecution is still present, removing 2023-04-12 02:54:12,332 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testRollbackAndDoubleExecution 2023-04-12 02:54:12,332 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testRollbackAndDoubleExecution' with timeout 60000ms 2023-04-12 02:54:12,332 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:54:12,332 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testRollbackAndDoubleExecution' starting 'acquire' stage 2023-04-12 02:54:12,342 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testRollbackAndDoubleExecution is still present, removing 2023-04-12 02:54:12,342 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testRollbackAndDoubleExecution 2023-04-12 02:54:12,342 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:54:12,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:54:12,342 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testRollbackAndDoubleExecution' with timeout 60000ms 2023-04-12 02:54:12,342 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:54:12,342 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testRollbackAndDoubleExecution' starting 'acquire' stage 2023-04-12 02:54:12,342 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testRollbackAndDoubleExecution' locally acquired 2023-04-12 02:54:12,343 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43091,1681268031804' joining acquired barrier for procedure (testRollbackAndDoubleExecution) in zk 2023-04-12 02:54:12,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-04-12 02:54:12,344 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 02:54:12,344 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 02:54:12,344 DEBUG [rs(jenkins-hbase4.apache.org,44679,1681268031772)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testRollbackAndDoubleExecution,,1681268051553.96cdae2f7638212730b694522b9a464a. 2023-04-12 02:54:12,344 DEBUG [rs(jenkins-hbase4.apache.org,44679,1681268031772)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testRollbackAndDoubleExecution,,1681268051553.96cdae2f7638212730b694522b9a464a. started... 2023-04-12 02:54:12,344 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-12 02:54:12,344 INFO [rs(jenkins-hbase4.apache.org,44679,1681268031772)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 96cdae2f7638212730b694522b9a464a 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-04-12 02:54:12,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 02:54:12,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-04-12 02:54:12,345 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testRollbackAndDoubleExecution 2023-04-12 02:54:12,345 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-12 02:54:12,346 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-12 02:54:12,346 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testRollbackAndDoubleExecution' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 02:54:12,348 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testRollbackAndDoubleExecution is still present, removing 2023-04-12 02:54:12,348 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testRollbackAndDoubleExecution 2023-04-12 02:54:12,348 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testRollbackAndDoubleExecution' with timeout 60000ms 2023-04-12 02:54:12,348 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:54:12,348 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testRollbackAndDoubleExecution' starting 'acquire' stage 2023-04-12 02:54:12,348 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testRollbackAndDoubleExecution' locally acquired 2023-04-12 02:54:12,348 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34579,1681268031719' joining acquired barrier for procedure (testRollbackAndDoubleExecution) in zk 2023-04-12 02:54:12,351 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-12 02:54:12,351 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-12 02:54:12,351 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testRollbackAndDoubleExecution' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 02:54:12,379 INFO [rs(jenkins-hbase4.apache.org,44679,1681268031772)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=19 (bloomFilter=true), to=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/.tmp/cf1/0693b446d83544e4a9700be8c15de027 2023-04-12 02:54:12,408 INFO [rs(jenkins-hbase4.apache.org,44679,1681268031772)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=19 (bloomFilter=true), to=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/.tmp/cf2/f66940ce4359474eac6c223cf1a8b486 2023-04-12 02:54:12,415 DEBUG [rs(jenkins-hbase4.apache.org,44679,1681268031772)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/.tmp/cf1/0693b446d83544e4a9700be8c15de027 as hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/cf1/0693b446d83544e4a9700be8c15de027 2023-04-12 02:54:12,420 INFO [rs(jenkins-hbase4.apache.org,44679,1681268031772)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/cf1/0693b446d83544e4a9700be8c15de027, entries=10, sequenceid=19, filesize=5.0 K 2023-04-12 02:54:12,422 DEBUG [rs(jenkins-hbase4.apache.org,44679,1681268031772)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/.tmp/cf2/f66940ce4359474eac6c223cf1a8b486 as hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/cf2/f66940ce4359474eac6c223cf1a8b486 2023-04-12 02:54:12,427 INFO [rs(jenkins-hbase4.apache.org,44679,1681268031772)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/cf2/f66940ce4359474eac6c223cf1a8b486, entries=10, sequenceid=19, filesize=5.0 K 2023-04-12 02:54:12,428 INFO [rs(jenkins-hbase4.apache.org,44679,1681268031772)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for 96cdae2f7638212730b694522b9a464a in 84ms, sequenceid=19, compaction requested=false 2023-04-12 02:54:12,428 DEBUG [rs(jenkins-hbase4.apache.org,44679,1681268031772)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 96cdae2f7638212730b694522b9a464a: 2023-04-12 02:54:12,428 DEBUG [rs(jenkins-hbase4.apache.org,44679,1681268031772)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testRollbackAndDoubleExecution,,1681268051553.96cdae2f7638212730b694522b9a464a. 2023-04-12 02:54:12,429 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 02:54:12,429 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 02:54:12,429 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:12,429 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testRollbackAndDoubleExecution' locally acquired 2023-04-12 02:54:12,429 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44679,1681268031772' joining acquired barrier for procedure (testRollbackAndDoubleExecution) in zk 2023-04-12 02:54:12,439 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:12,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:12,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 02:54:12,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 02:54:12,439 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-12 02:54:12,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 02:54:12,440 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-12 02:54:12,440 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testRollbackAndDoubleExecution' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 02:54:12,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 02:54:12,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRollbackAndDoubleExecution 2023-04-12 02:54:12,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:12,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:12,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:12,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 02:54:12,442 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44679,1681268031772' joining acquired barrier for procedure 'testRollbackAndDoubleExecution' on coordinator 2023-04-12 02:54:12,442 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6ae2946[Count = 0] remaining members to acquire global barrier 2023-04-12 02:54:12,442 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testRollbackAndDoubleExecution' starting 'in-barrier' execution. 2023-04-12 02:54:12,442 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-12 02:54:12,443 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-12 02:54:12,443 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-12 02:54:12,443 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-12 02:54:12,443 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-12 02:54:12,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-12 02:54:12,443 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-12 02:54:12,444 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testRollbackAndDoubleExecution' received 'reached' from coordinator. 2023-04-12 02:54:12,444 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-12 02:54:12,444 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testRollbackAndDoubleExecution' locally completed 2023-04-12 02:54:12,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-12 02:54:12,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-12 02:54:12,444 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testRollbackAndDoubleExecution' received 'reached' from coordinator. 2023-04-12 02:54:12,444 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:12,444 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 02:54:12,444 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testRollbackAndDoubleExecution' completed for member 'jenkins-hbase4.apache.org,34579,1681268031719' in zk 2023-04-12 02:54:12,444 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testRollbackAndDoubleExecution' locally completed 2023-04-12 02:54:12,444 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testRollbackAndDoubleExecution' received 'reached' from coordinator. 2023-04-12 02:54:12,444 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testRollbackAndDoubleExecution' locally completed 2023-04-12 02:54:12,444 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testRollbackAndDoubleExecution' completed for member 'jenkins-hbase4.apache.org,43091,1681268031804' in zk 2023-04-12 02:54:12,444 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testRollbackAndDoubleExecution' completed for member 'jenkins-hbase4.apache.org,44679,1681268031772' in zk 2023-04-12 02:54:12,446 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testRollbackAndDoubleExecution' has notified controller of completion 2023-04-12 02:54:12,446 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:54:12,446 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testRollbackAndDoubleExecution' completed. 2023-04-12 02:54:12,450 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:12,450 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testRollbackAndDoubleExecution' has notified controller of completion 2023-04-12 02:54:12,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:12,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 02:54:12,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 02:54:12,450 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testRollbackAndDoubleExecution' has notified controller of completion 2023-04-12 02:54:12,450 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:54:12,450 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:54:12,450 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testRollbackAndDoubleExecution' completed. 2023-04-12 02:54:12,450 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testRollbackAndDoubleExecution' completed. 2023-04-12 02:54:12,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 02:54:12,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 02:54:12,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRollbackAndDoubleExecution 2023-04-12 02:54:12,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:12,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:12,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:12,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 02:54:12,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRollbackAndDoubleExecution 2023-04-12 02:54:12,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:12,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:12,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:12,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testRollbackAndDoubleExecution' member 'jenkins-hbase4.apache.org,44679,1681268031772': 2023-04-12 02:54:12,454 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44679,1681268031772' released barrier for procedure'testRollbackAndDoubleExecution', counting down latch. Waiting for 0 more 2023-04-12 02:54:12,454 INFO [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testRollbackAndDoubleExecution' execution completed 2023-04-12 02:54:12,454 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 02:54:12,454 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 02:54:12,454 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testRollbackAndDoubleExecution 2023-04-12 02:54:12,454 INFO [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testRollbackAndDoubleExecutionincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 02:54:12,456 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-12 02:54:12,456 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-12 02:54:12,456 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-12 02:54:12,456 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-12 02:54:12,456 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:54:12,456 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-12 02:54:12,456 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:54:12,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-12 02:54:12,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 02:54:12,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 02:54:12,456 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-12 02:54:12,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-12 02:54:12,456 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-12 02:54:12,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-12 02:54:12,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-12 02:54:12,456 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:54:12,457 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:54:12,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:54:12,457 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:54:12,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:54:12,457 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:54:12,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:54:12,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 02:54:12,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-12 02:54:12,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-12 02:54:12,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRollbackAndDoubleExecution 2023-04-12 02:54:12,458 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:12,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 02:54:12,459 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:12,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRollbackAndDoubleExecution 2023-04-12 02:54:12,459 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:12,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:12,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:12,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:12,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 02:54:12,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRollbackAndDoubleExecution 2023-04-12 02:54:12,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-12 02:54:12,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:12,462 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:12,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:12,462 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:12,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:12,463 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:12,466 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:12,466 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:54:12,466 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:54:12,466 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-04-12 02:54:12,466 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:12,466 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:54:12,466 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:54:12,466 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testRollbackAndDoubleExecution' 2023-04-12 02:54:12,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:54:12,466 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 02:54:12,466 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:54:12,467 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:54:12,466 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:12,467 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-04-12 02:54:12,467 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:12,467 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-12 02:54:12,467 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:12,467 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:12,467 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-12 02:54:12,467 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-12 02:54:12,466 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:54:12,467 DEBUG [Listener at localhost/46667] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testRollbackAndDoubleExecution'' to complete. (max 20000 ms per retry) 2023-04-12 02:54:12,467 DEBUG [Listener at localhost/46667] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 02:54:12,467 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:54:12,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:54:12,467 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:54:12,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:54:12,467 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:54:12,468 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:54:12,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:54:12,470 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:54:12,471 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:54:12,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:54:12,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:54:12,568 DEBUG [Listener at localhost/46667] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 02:54:12,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 02:54:12,592 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 02:54:12,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] procedure.ProcedureCoordinator(143): Procedure testRollbackAndDoubleExecution was in running list but was completed. Accepting new attempt. 2023-04-12 02:54:12,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] procedure.ProcedureCoordinator(165): Submitting procedure testRollbackAndDoubleExecution 2023-04-12 02:54:12,596 INFO [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testRollbackAndDoubleExecution' 2023-04-12 02:54:12,596 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:54:12,597 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testRollbackAndDoubleExecution' starting 'acquire' 2023-04-12 02:54:12,597 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testRollbackAndDoubleExecution', kicking off acquire phase on members. 2023-04-12 02:54:12,597 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-12 02:54:12,598 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-04-12 02:54:12,600 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:54:12,600 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:12,600 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:54:12,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:54:12,600 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:54:12,600 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:54:12,600 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:54:12,600 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:54:12,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:54:12,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:54:12,600 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:12,601 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 02:54:12,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-04-12 02:54:12,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-04-12 02:54:12,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-04-12 02:54:12,601 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-12 02:54:12,601 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-12 02:54:12,602 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-12 02:54:12,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 02:54:12,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-04-12 02:54:12,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 02:54:12,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-04-12 02:54:12,602 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testRollbackAndDoubleExecution 2023-04-12 02:54:12,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 02:54:12,602 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testRollbackAndDoubleExecution 2023-04-12 02:54:12,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-04-12 02:54:12,603 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testRollbackAndDoubleExecution 2023-04-12 02:54:12,603 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testRollbackAndDoubleExecution is still present, removing 2023-04-12 02:54:12,603 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testRollbackAndDoubleExecution 2023-04-12 02:54:12,603 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testRollbackAndDoubleExecution' with timeout 60000ms 2023-04-12 02:54:12,603 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testRollbackAndDoubleExecution is still present, removing 2023-04-12 02:54:12,603 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testRollbackAndDoubleExecution 2023-04-12 02:54:12,603 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:54:12,603 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testRollbackAndDoubleExecution' with timeout 60000ms 2023-04-12 02:54:12,603 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testRollbackAndDoubleExecution is still present, removing 2023-04-12 02:54:12,603 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testRollbackAndDoubleExecution 2023-04-12 02:54:12,603 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testRollbackAndDoubleExecution' with timeout 60000ms 2023-04-12 02:54:12,603 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:54:12,603 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testRollbackAndDoubleExecution' starting 'acquire' stage 2023-04-12 02:54:12,603 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testRollbackAndDoubleExecution' locally acquired 2023-04-12 02:54:12,603 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43091,1681268031804' joining acquired barrier for procedure (testRollbackAndDoubleExecution) in zk 2023-04-12 02:54:12,603 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:54:12,603 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testRollbackAndDoubleExecution' starting 'acquire' stage 2023-04-12 02:54:12,603 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testRollbackAndDoubleExecution' starting 'acquire' stage 2023-04-12 02:54:12,604 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testRollbackAndDoubleExecution' locally acquired 2023-04-12 02:54:12,604 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34579,1681268031719' joining acquired barrier for procedure (testRollbackAndDoubleExecution) in zk 2023-04-12 02:54:12,607 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-12 02:54:12,608 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-12 02:54:12,608 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testRollbackAndDoubleExecution' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 02:54:12,608 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-12 02:54:12,608 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-12 02:54:12,608 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testRollbackAndDoubleExecution' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 02:54:12,611 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 02:54:12,611 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 02:54:12,611 DEBUG [rs(jenkins-hbase4.apache.org,44679,1681268031772)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testRollbackAndDoubleExecution,,1681268051553.96cdae2f7638212730b694522b9a464a. 2023-04-12 02:54:12,611 DEBUG [rs(jenkins-hbase4.apache.org,44679,1681268031772)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testRollbackAndDoubleExecution,,1681268051553.96cdae2f7638212730b694522b9a464a. started... 2023-04-12 02:54:12,612 INFO [rs(jenkins-hbase4.apache.org,44679,1681268031772)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 96cdae2f7638212730b694522b9a464a 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-04-12 02:54:13,028 INFO [rs(jenkins-hbase4.apache.org,44679,1681268031772)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=32 (bloomFilter=true), to=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/.tmp/cf1/091b8b6c71524b5ea1657e72778f8478 2023-04-12 02:54:13,052 INFO [rs(jenkins-hbase4.apache.org,44679,1681268031772)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=32 (bloomFilter=true), to=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/.tmp/cf2/6c612c6ffd3d42eab35c5ae09327ecf8 2023-04-12 02:54:13,067 DEBUG [rs(jenkins-hbase4.apache.org,44679,1681268031772)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/.tmp/cf1/091b8b6c71524b5ea1657e72778f8478 as hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/cf1/091b8b6c71524b5ea1657e72778f8478 2023-04-12 02:54:13,074 INFO [rs(jenkins-hbase4.apache.org,44679,1681268031772)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/cf1/091b8b6c71524b5ea1657e72778f8478, entries=10, sequenceid=32, filesize=5.0 K 2023-04-12 02:54:13,075 DEBUG [rs(jenkins-hbase4.apache.org,44679,1681268031772)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/.tmp/cf2/6c612c6ffd3d42eab35c5ae09327ecf8 as hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/cf2/6c612c6ffd3d42eab35c5ae09327ecf8 2023-04-12 02:54:13,080 INFO [rs(jenkins-hbase4.apache.org,44679,1681268031772)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/cf2/6c612c6ffd3d42eab35c5ae09327ecf8, entries=10, sequenceid=32, filesize=5.0 K 2023-04-12 02:54:13,081 INFO [rs(jenkins-hbase4.apache.org,44679,1681268031772)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for 96cdae2f7638212730b694522b9a464a in 469ms, sequenceid=32, compaction requested=true 2023-04-12 02:54:13,082 DEBUG [rs(jenkins-hbase4.apache.org,44679,1681268031772)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 96cdae2f7638212730b694522b9a464a: 2023-04-12 02:54:13,082 DEBUG [rs(jenkins-hbase4.apache.org,44679,1681268031772)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testRollbackAndDoubleExecution,,1681268051553.96cdae2f7638212730b694522b9a464a. 2023-04-12 02:54:13,082 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 02:54:13,082 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 02:54:13,082 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:13,082 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testRollbackAndDoubleExecution' locally acquired 2023-04-12 02:54:13,082 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44679,1681268031772' joining acquired barrier for procedure (testRollbackAndDoubleExecution) in zk 2023-04-12 02:54:13,084 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:13,084 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-12 02:54:13,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:13,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 02:54:13,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 02:54:13,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 02:54:13,084 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-12 02:54:13,084 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testRollbackAndDoubleExecution' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 02:54:13,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 02:54:13,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRollbackAndDoubleExecution 2023-04-12 02:54:13,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:13,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:13,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:13,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 02:54:13,087 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44679,1681268031772' joining acquired barrier for procedure 'testRollbackAndDoubleExecution' on coordinator 2023-04-12 02:54:13,088 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1f20b6[Count = 0] remaining members to acquire global barrier 2023-04-12 02:54:13,088 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testRollbackAndDoubleExecution' starting 'in-barrier' execution. 2023-04-12 02:54:13,088 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-12 02:54:13,089 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-12 02:54:13,089 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-12 02:54:13,089 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-12 02:54:13,089 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-12 02:54:13,089 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-12 02:54:13,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-12 02:54:13,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-12 02:54:13,089 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-12 02:54:13,089 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testRollbackAndDoubleExecution' received 'reached' from coordinator. 2023-04-12 02:54:13,090 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testRollbackAndDoubleExecution' locally completed 2023-04-12 02:54:13,090 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testRollbackAndDoubleExecution' completed for member 'jenkins-hbase4.apache.org,43091,1681268031804' in zk 2023-04-12 02:54:13,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-12 02:54:13,089 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:13,090 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 02:54:13,089 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testRollbackAndDoubleExecution' received 'reached' from coordinator. 2023-04-12 02:54:13,090 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testRollbackAndDoubleExecution' locally completed 2023-04-12 02:54:13,090 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testRollbackAndDoubleExecution' completed for member 'jenkins-hbase4.apache.org,44679,1681268031772' in zk 2023-04-12 02:54:13,090 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testRollbackAndDoubleExecution' received 'reached' from coordinator. 2023-04-12 02:54:13,090 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testRollbackAndDoubleExecution' locally completed 2023-04-12 02:54:13,090 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testRollbackAndDoubleExecution' completed for member 'jenkins-hbase4.apache.org,34579,1681268031719' in zk 2023-04-12 02:54:13,092 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testRollbackAndDoubleExecution' has notified controller of completion 2023-04-12 02:54:13,092 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:54:13,092 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testRollbackAndDoubleExecution' completed. 2023-04-12 02:54:13,093 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testRollbackAndDoubleExecution' has notified controller of completion 2023-04-12 02:54:13,093 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:54:13,093 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testRollbackAndDoubleExecution' completed. 2023-04-12 02:54:13,097 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testRollbackAndDoubleExecution' has notified controller of completion 2023-04-12 02:54:13,097 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:54:13,097 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:13,097 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testRollbackAndDoubleExecution' completed. 2023-04-12 02:54:13,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:13,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 02:54:13,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 02:54:13,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 02:54:13,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 02:54:13,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRollbackAndDoubleExecution 2023-04-12 02:54:13,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:13,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:13,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:13,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 02:54:13,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRollbackAndDoubleExecution 2023-04-12 02:54:13,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:13,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:13,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:13,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testRollbackAndDoubleExecution' member 'jenkins-hbase4.apache.org,44679,1681268031772': 2023-04-12 02:54:13,102 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44679,1681268031772' released barrier for procedure'testRollbackAndDoubleExecution', counting down latch. Waiting for 0 more 2023-04-12 02:54:13,102 INFO [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testRollbackAndDoubleExecution' execution completed 2023-04-12 02:54:13,102 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 02:54:13,102 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 02:54:13,102 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testRollbackAndDoubleExecution 2023-04-12 02:54:13,102 INFO [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testRollbackAndDoubleExecutionincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 02:54:13,103 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-12 02:54:13,103 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:54:13,104 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-12 02:54:13,104 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-12 02:54:13,104 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-12 02:54:13,104 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-12 02:54:13,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-12 02:54:13,104 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-12 02:54:13,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 02:54:13,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 02:54:13,104 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:54:13,104 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-12 02:54:13,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-12 02:54:13,104 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:54:13,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 02:54:13,105 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:54:13,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:54:13,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-12 02:54:13,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-12 02:54:13,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRollbackAndDoubleExecution 2023-04-12 02:54:13,105 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:13,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-12 02:54:13,112 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:54:13,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:54:13,112 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:13,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 02:54:13,112 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:54:13,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:54:13,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-12 02:54:13,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRollbackAndDoubleExecution 2023-04-12 02:54:13,112 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:13,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-12 02:54:13,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:13,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:13,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:13,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 02:54:13,115 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:13,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRollbackAndDoubleExecution 2023-04-12 02:54:13,115 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:13,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:13,116 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:13,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:13,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:13,126 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:54:13,126 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:13,126 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-04-12 02:54:13,126 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:13,126 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:54:13,126 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:54:13,126 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:54:13,126 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:54:13,126 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:54:13,126 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testRollbackAndDoubleExecution' 2023-04-12 02:54:13,126 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 02:54:13,126 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:13,126 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:54:13,126 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:54:13,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:54:13,126 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-04-12 02:54:13,126 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:54:13,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:54:13,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:54:13,126 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:54:13,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:54:13,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:54:13,126 DEBUG [Listener at localhost/46667] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testRollbackAndDoubleExecution'' to complete. (max 20000 ms per retry) 2023-04-12 02:54:13,127 DEBUG [Listener at localhost/46667] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 02:54:13,126 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:13,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:54:13,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:54:13,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:54:13,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:54:13,127 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-12 02:54:13,127 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:13,127 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:13,127 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-12 02:54:13,127 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-12 02:54:13,227 DEBUG [Listener at localhost/46667] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 02:54:13,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 02:54:13,245 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 02:54:13,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] procedure.ProcedureCoordinator(143): Procedure testRollbackAndDoubleExecution was in running list but was completed. Accepting new attempt. 2023-04-12 02:54:13,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] procedure.ProcedureCoordinator(165): Submitting procedure testRollbackAndDoubleExecution 2023-04-12 02:54:13,247 INFO [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testRollbackAndDoubleExecution' 2023-04-12 02:54:13,247 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:54:13,247 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testRollbackAndDoubleExecution' starting 'acquire' 2023-04-12 02:54:13,247 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testRollbackAndDoubleExecution', kicking off acquire phase on members. 2023-04-12 02:54:13,248 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-12 02:54:13,248 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-04-12 02:54:13,250 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:54:13,250 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:54:13,250 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:13,250 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:54:13,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:54:13,250 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:54:13,250 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:54:13,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:54:13,250 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:54:13,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:54:13,250 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:13,250 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 02:54:13,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-04-12 02:54:13,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-04-12 02:54:13,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-04-12 02:54:13,251 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-12 02:54:13,251 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-12 02:54:13,251 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-12 02:54:13,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 02:54:13,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-04-12 02:54:13,251 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testRollbackAndDoubleExecution 2023-04-12 02:54:13,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 02:54:13,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 02:54:13,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-04-12 02:54:13,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-04-12 02:54:13,251 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testRollbackAndDoubleExecution 2023-04-12 02:54:13,251 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testRollbackAndDoubleExecution is still present, removing 2023-04-12 02:54:13,251 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testRollbackAndDoubleExecution 2023-04-12 02:54:13,252 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testRollbackAndDoubleExecution 2023-04-12 02:54:13,252 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testRollbackAndDoubleExecution is still present, removing 2023-04-12 02:54:13,252 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testRollbackAndDoubleExecution' with timeout 60000ms 2023-04-12 02:54:13,252 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:54:13,252 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testRollbackAndDoubleExecution 2023-04-12 02:54:13,252 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testRollbackAndDoubleExecution' starting 'acquire' stage 2023-04-12 02:54:13,252 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testRollbackAndDoubleExecution' with timeout 60000ms 2023-04-12 02:54:13,252 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testRollbackAndDoubleExecution is still present, removing 2023-04-12 02:54:13,252 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:54:13,252 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testRollbackAndDoubleExecution' locally acquired 2023-04-12 02:54:13,252 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testRollbackAndDoubleExecution' starting 'acquire' stage 2023-04-12 02:54:13,252 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testRollbackAndDoubleExecution' locally acquired 2023-04-12 02:54:13,252 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34579,1681268031719' joining acquired barrier for procedure (testRollbackAndDoubleExecution) in zk 2023-04-12 02:54:13,252 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testRollbackAndDoubleExecution 2023-04-12 02:54:13,252 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43091,1681268031804' joining acquired barrier for procedure (testRollbackAndDoubleExecution) in zk 2023-04-12 02:54:13,252 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testRollbackAndDoubleExecution' with timeout 60000ms 2023-04-12 02:54:13,252 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:54:13,252 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testRollbackAndDoubleExecution' starting 'acquire' stage 2023-04-12 02:54:13,253 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 02:54:13,253 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 02:54:13,253 DEBUG [rs(jenkins-hbase4.apache.org,44679,1681268031772)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testRollbackAndDoubleExecution,,1681268051553.96cdae2f7638212730b694522b9a464a. 2023-04-12 02:54:13,253 DEBUG [rs(jenkins-hbase4.apache.org,44679,1681268031772)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testRollbackAndDoubleExecution,,1681268051553.96cdae2f7638212730b694522b9a464a. started... 2023-04-12 02:54:13,253 INFO [rs(jenkins-hbase4.apache.org,44679,1681268031772)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 96cdae2f7638212730b694522b9a464a 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-04-12 02:54:13,255 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-12 02:54:13,255 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-12 02:54:13,255 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-12 02:54:13,255 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testRollbackAndDoubleExecution' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 02:54:13,255 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-12 02:54:13,255 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testRollbackAndDoubleExecution' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 02:54:13,269 INFO [rs(jenkins-hbase4.apache.org,44679,1681268031772)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=45 (bloomFilter=true), to=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/.tmp/cf1/177232c91ccf4f5d91bf18c2d4d5022f 2023-04-12 02:54:13,305 INFO [rs(jenkins-hbase4.apache.org,44679,1681268031772)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=45 (bloomFilter=true), to=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/.tmp/cf2/11c4addd69d049beb33f17d53e40ab23 2023-04-12 02:54:13,310 DEBUG [rs(jenkins-hbase4.apache.org,44679,1681268031772)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/.tmp/cf1/177232c91ccf4f5d91bf18c2d4d5022f as hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/cf1/177232c91ccf4f5d91bf18c2d4d5022f 2023-04-12 02:54:13,317 INFO [rs(jenkins-hbase4.apache.org,44679,1681268031772)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/cf1/177232c91ccf4f5d91bf18c2d4d5022f, entries=10, sequenceid=45, filesize=5.0 K 2023-04-12 02:54:13,317 DEBUG [rs(jenkins-hbase4.apache.org,44679,1681268031772)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/.tmp/cf2/11c4addd69d049beb33f17d53e40ab23 as hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/cf2/11c4addd69d049beb33f17d53e40ab23 2023-04-12 02:54:13,325 INFO [rs(jenkins-hbase4.apache.org,44679,1681268031772)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/cf2/11c4addd69d049beb33f17d53e40ab23, entries=10, sequenceid=45, filesize=5.0 K 2023-04-12 02:54:13,326 INFO [rs(jenkins-hbase4.apache.org,44679,1681268031772)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for 96cdae2f7638212730b694522b9a464a in 73ms, sequenceid=45, compaction requested=true 2023-04-12 02:54:13,326 DEBUG [rs(jenkins-hbase4.apache.org,44679,1681268031772)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 96cdae2f7638212730b694522b9a464a: 2023-04-12 02:54:13,326 DEBUG [rs(jenkins-hbase4.apache.org,44679,1681268031772)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testRollbackAndDoubleExecution,,1681268051553.96cdae2f7638212730b694522b9a464a. 2023-04-12 02:54:13,326 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 02:54:13,326 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 02:54:13,326 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:13,326 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testRollbackAndDoubleExecution' locally acquired 2023-04-12 02:54:13,326 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44679,1681268031772' joining acquired barrier for procedure (testRollbackAndDoubleExecution) in zk 2023-04-12 02:54:13,336 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:13,336 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-12 02:54:13,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:13,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 02:54:13,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 02:54:13,336 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-12 02:54:13,336 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testRollbackAndDoubleExecution' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 02:54:13,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 02:54:13,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 02:54:13,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRollbackAndDoubleExecution 2023-04-12 02:54:13,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:13,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:13,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:13,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 02:54:13,339 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44679,1681268031772' joining acquired barrier for procedure 'testRollbackAndDoubleExecution' on coordinator 2023-04-12 02:54:13,339 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@76a41180[Count = 0] remaining members to acquire global barrier 2023-04-12 02:54:13,339 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testRollbackAndDoubleExecution' starting 'in-barrier' execution. 2023-04-12 02:54:13,339 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-12 02:54:13,341 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-12 02:54:13,341 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-12 02:54:13,341 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-12 02:54:13,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-12 02:54:13,341 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-12 02:54:13,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-12 02:54:13,342 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testRollbackAndDoubleExecution' received 'reached' from coordinator. 2023-04-12 02:54:13,342 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testRollbackAndDoubleExecution' locally completed 2023-04-12 02:54:13,342 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-12 02:54:13,342 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:13,342 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-12 02:54:13,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-12 02:54:13,342 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testRollbackAndDoubleExecution' completed for member 'jenkins-hbase4.apache.org,34579,1681268031719' in zk 2023-04-12 02:54:13,342 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testRollbackAndDoubleExecution' received 'reached' from coordinator. 2023-04-12 02:54:13,342 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testRollbackAndDoubleExecution' locally completed 2023-04-12 02:54:13,342 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testRollbackAndDoubleExecution' completed for member 'jenkins-hbase4.apache.org,43091,1681268031804' in zk 2023-04-12 02:54:13,342 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 02:54:13,342 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testRollbackAndDoubleExecution' received 'reached' from coordinator. 2023-04-12 02:54:13,343 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testRollbackAndDoubleExecution' locally completed 2023-04-12 02:54:13,343 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testRollbackAndDoubleExecution' completed for member 'jenkins-hbase4.apache.org,44679,1681268031772' in zk 2023-04-12 02:54:13,343 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testRollbackAndDoubleExecution' has notified controller of completion 2023-04-12 02:54:13,343 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:54:13,344 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testRollbackAndDoubleExecution' completed. 2023-04-12 02:54:13,344 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testRollbackAndDoubleExecution' has notified controller of completion 2023-04-12 02:54:13,344 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:54:13,344 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testRollbackAndDoubleExecution' completed. 2023-04-12 02:54:13,345 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:13,345 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testRollbackAndDoubleExecution' has notified controller of completion 2023-04-12 02:54:13,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:13,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 02:54:13,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 02:54:13,345 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:54:13,345 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testRollbackAndDoubleExecution' completed. 2023-04-12 02:54:13,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 02:54:13,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 02:54:13,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRollbackAndDoubleExecution 2023-04-12 02:54:13,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:13,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:13,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:13,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 02:54:13,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRollbackAndDoubleExecution 2023-04-12 02:54:13,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:13,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:13,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:13,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testRollbackAndDoubleExecution' member 'jenkins-hbase4.apache.org,44679,1681268031772': 2023-04-12 02:54:13,350 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44679,1681268031772' released barrier for procedure'testRollbackAndDoubleExecution', counting down latch. Waiting for 0 more 2023-04-12 02:54:13,350 INFO [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testRollbackAndDoubleExecution' execution completed 2023-04-12 02:54:13,350 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 02:54:13,350 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 02:54:13,350 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testRollbackAndDoubleExecution 2023-04-12 02:54:13,350 INFO [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testRollbackAndDoubleExecutionincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 02:54:13,354 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-12 02:54:13,354 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-12 02:54:13,354 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-12 02:54:13,354 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-12 02:54:13,354 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-12 02:54:13,354 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:54:13,354 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:54:13,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-12 02:54:13,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 02:54:13,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 02:54:13,354 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-12 02:54:13,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-12 02:54:13,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-12 02:54:13,354 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-12 02:54:13,354 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:54:13,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-12 02:54:13,355 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:13,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:54:13,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:54:13,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:54:13,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:54:13,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:54:13,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:54:13,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-12 02:54:13,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-12 02:54:13,356 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:13,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 02:54:13,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-12 02:54:13,356 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:13,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRollbackAndDoubleExecution 2023-04-12 02:54:13,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 02:54:13,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRollbackAndDoubleExecution 2023-04-12 02:54:13,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:13,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:13,359 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:13,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:13,359 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:13,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 02:54:13,360 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:13,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRollbackAndDoubleExecution 2023-04-12 02:54:13,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:13,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:13,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:13,364 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:54:13,364 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:54:13,364 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:54:13,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:54:13,364 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:54:13,365 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:54:13,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:54:13,365 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:54:13,365 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:54:13,365 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:54:13,365 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:54:13,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:54:13,365 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:13,365 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:54:13,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:54:13,365 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-04-12 02:54:13,365 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:13,365 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:13,365 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-04-12 02:54:13,366 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:54:13,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:54:13,365 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:54:13,366 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testRollbackAndDoubleExecution' 2023-04-12 02:54:13,366 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 02:54:13,366 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:13,366 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-12 02:54:13,366 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:13,366 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:13,366 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-12 02:54:13,366 DEBUG [Listener at localhost/46667] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testRollbackAndDoubleExecution'' to complete. (max 20000 ms per retry) 2023-04-12 02:54:13,366 DEBUG [Listener at localhost/46667] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 02:54:13,366 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-12 02:54:13,366 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:54:13,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:54:13,466 DEBUG [Listener at localhost/46667] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 02:54:13,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 02:54:13,493 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 02:54:13,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] procedure.ProcedureCoordinator(143): Procedure testRollbackAndDoubleExecution was in running list but was completed. Accepting new attempt. 2023-04-12 02:54:13,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] procedure.ProcedureCoordinator(165): Submitting procedure testRollbackAndDoubleExecution 2023-04-12 02:54:13,495 INFO [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testRollbackAndDoubleExecution' 2023-04-12 02:54:13,495 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:54:13,495 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testRollbackAndDoubleExecution' starting 'acquire' 2023-04-12 02:54:13,495 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testRollbackAndDoubleExecution', kicking off acquire phase on members. 2023-04-12 02:54:13,496 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-12 02:54:13,496 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-04-12 02:54:13,497 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:13,497 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:54:13,497 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:54:13,497 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:54:13,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:54:13,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:54:13,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:54:13,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:54:13,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:54:13,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:54:13,498 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:13,498 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 02:54:13,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-04-12 02:54:13,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-04-12 02:54:13,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-04-12 02:54:13,499 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-12 02:54:13,499 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-12 02:54:13,499 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-12 02:54:13,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 02:54:13,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-04-12 02:54:13,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 02:54:13,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-04-12 02:54:13,499 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testRollbackAndDoubleExecution 2023-04-12 02:54:13,499 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testRollbackAndDoubleExecution 2023-04-12 02:54:13,500 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testRollbackAndDoubleExecution is still present, removing 2023-04-12 02:54:13,500 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testRollbackAndDoubleExecution is still present, removing 2023-04-12 02:54:13,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 02:54:13,500 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testRollbackAndDoubleExecution 2023-04-12 02:54:13,500 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testRollbackAndDoubleExecution 2023-04-12 02:54:13,500 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testRollbackAndDoubleExecution' with timeout 60000ms 2023-04-12 02:54:13,500 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:54:13,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-04-12 02:54:13,500 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testRollbackAndDoubleExecution' starting 'acquire' stage 2023-04-12 02:54:13,500 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testRollbackAndDoubleExecution' locally acquired 2023-04-12 02:54:13,500 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testRollbackAndDoubleExecution' with timeout 60000ms 2023-04-12 02:54:13,500 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43091,1681268031804' joining acquired barrier for procedure (testRollbackAndDoubleExecution) in zk 2023-04-12 02:54:13,500 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testRollbackAndDoubleExecution 2023-04-12 02:54:13,500 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:54:13,501 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testRollbackAndDoubleExecution' starting 'acquire' stage 2023-04-12 02:54:13,501 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testRollbackAndDoubleExecution' locally acquired 2023-04-12 02:54:13,501 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34579,1681268031719' joining acquired barrier for procedure (testRollbackAndDoubleExecution) in zk 2023-04-12 02:54:13,501 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testRollbackAndDoubleExecution is still present, removing 2023-04-12 02:54:13,501 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testRollbackAndDoubleExecution 2023-04-12 02:54:13,501 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testRollbackAndDoubleExecution' with timeout 60000ms 2023-04-12 02:54:13,501 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:54:13,501 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testRollbackAndDoubleExecution' starting 'acquire' stage 2023-04-12 02:54:13,502 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-12 02:54:13,504 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-12 02:54:13,504 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-12 02:54:13,504 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testRollbackAndDoubleExecution' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 02:54:13,504 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-12 02:54:13,504 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testRollbackAndDoubleExecution' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 02:54:13,506 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 02:54:13,506 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 02:54:13,506 DEBUG [rs(jenkins-hbase4.apache.org,44679,1681268031772)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testRollbackAndDoubleExecution,,1681268051553.96cdae2f7638212730b694522b9a464a. 2023-04-12 02:54:13,506 DEBUG [rs(jenkins-hbase4.apache.org,44679,1681268031772)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testRollbackAndDoubleExecution,,1681268051553.96cdae2f7638212730b694522b9a464a. started... 2023-04-12 02:54:13,506 INFO [rs(jenkins-hbase4.apache.org,44679,1681268031772)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 96cdae2f7638212730b694522b9a464a 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-04-12 02:54:13,527 INFO [rs(jenkins-hbase4.apache.org,44679,1681268031772)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=58 (bloomFilter=true), to=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/.tmp/cf1/0864a86174d140c0a957973bc4f9f7dc 2023-04-12 02:54:13,543 INFO [rs(jenkins-hbase4.apache.org,44679,1681268031772)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=58 (bloomFilter=true), to=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/.tmp/cf2/cd4efe4b5ad64256947de9915b5f6552 2023-04-12 02:54:13,549 DEBUG [rs(jenkins-hbase4.apache.org,44679,1681268031772)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/.tmp/cf1/0864a86174d140c0a957973bc4f9f7dc as hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/cf1/0864a86174d140c0a957973bc4f9f7dc 2023-04-12 02:54:13,554 INFO [rs(jenkins-hbase4.apache.org,44679,1681268031772)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/cf1/0864a86174d140c0a957973bc4f9f7dc, entries=10, sequenceid=58, filesize=5.0 K 2023-04-12 02:54:13,555 DEBUG [rs(jenkins-hbase4.apache.org,44679,1681268031772)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/.tmp/cf2/cd4efe4b5ad64256947de9915b5f6552 as hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/cf2/cd4efe4b5ad64256947de9915b5f6552 2023-04-12 02:54:13,559 INFO [rs(jenkins-hbase4.apache.org,44679,1681268031772)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/cf2/cd4efe4b5ad64256947de9915b5f6552, entries=10, sequenceid=58, filesize=5.0 K 2023-04-12 02:54:13,560 INFO [rs(jenkins-hbase4.apache.org,44679,1681268031772)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for 96cdae2f7638212730b694522b9a464a in 54ms, sequenceid=58, compaction requested=true 2023-04-12 02:54:13,561 DEBUG [rs(jenkins-hbase4.apache.org,44679,1681268031772)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 96cdae2f7638212730b694522b9a464a: 2023-04-12 02:54:13,561 DEBUG [rs(jenkins-hbase4.apache.org,44679,1681268031772)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testRollbackAndDoubleExecution,,1681268051553.96cdae2f7638212730b694522b9a464a. 2023-04-12 02:54:13,561 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 02:54:13,561 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 02:54:13,561 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:13,561 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testRollbackAndDoubleExecution' locally acquired 2023-04-12 02:54:13,561 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44679,1681268031772' joining acquired barrier for procedure (testRollbackAndDoubleExecution) in zk 2023-04-12 02:54:13,563 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:13,563 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-12 02:54:13,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:13,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 02:54:13,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 02:54:13,563 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-12 02:54:13,563 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testRollbackAndDoubleExecution' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 02:54:13,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 02:54:13,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 02:54:13,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRollbackAndDoubleExecution 2023-04-12 02:54:13,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:13,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:13,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:13,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 02:54:13,565 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44679,1681268031772' joining acquired barrier for procedure 'testRollbackAndDoubleExecution' on coordinator 2023-04-12 02:54:13,565 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@57756e61[Count = 0] remaining members to acquire global barrier 2023-04-12 02:54:13,565 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testRollbackAndDoubleExecution' starting 'in-barrier' execution. 2023-04-12 02:54:13,565 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-12 02:54:13,566 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-12 02:54:13,566 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-12 02:54:13,566 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-12 02:54:13,566 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-12 02:54:13,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-12 02:54:13,567 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testRollbackAndDoubleExecution' received 'reached' from coordinator. 2023-04-12 02:54:13,567 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testRollbackAndDoubleExecution' locally completed 2023-04-12 02:54:13,566 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-12 02:54:13,567 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-12 02:54:13,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-12 02:54:13,567 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testRollbackAndDoubleExecution' completed for member 'jenkins-hbase4.apache.org,44679,1681268031772' in zk 2023-04-12 02:54:13,567 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:13,567 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testRollbackAndDoubleExecution' received 'reached' from coordinator. 2023-04-12 02:54:13,567 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testRollbackAndDoubleExecution' locally completed 2023-04-12 02:54:13,567 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testRollbackAndDoubleExecution' completed for member 'jenkins-hbase4.apache.org,43091,1681268031804' in zk 2023-04-12 02:54:13,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-12 02:54:13,567 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 02:54:13,567 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testRollbackAndDoubleExecution' received 'reached' from coordinator. 2023-04-12 02:54:13,567 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testRollbackAndDoubleExecution' locally completed 2023-04-12 02:54:13,567 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testRollbackAndDoubleExecution' completed for member 'jenkins-hbase4.apache.org,34579,1681268031719' in zk 2023-04-12 02:54:13,568 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:13,569 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testRollbackAndDoubleExecution' has notified controller of completion 2023-04-12 02:54:13,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:13,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 02:54:13,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 02:54:13,569 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:54:13,569 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testRollbackAndDoubleExecution' completed. 2023-04-12 02:54:13,573 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testRollbackAndDoubleExecution' has notified controller of completion 2023-04-12 02:54:13,573 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testRollbackAndDoubleExecution' has notified controller of completion 2023-04-12 02:54:13,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 02:54:13,573 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:54:13,573 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:54:13,573 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testRollbackAndDoubleExecution' completed. 2023-04-12 02:54:13,573 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testRollbackAndDoubleExecution' completed. 2023-04-12 02:54:13,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 02:54:13,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRollbackAndDoubleExecution 2023-04-12 02:54:13,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:13,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:13,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:13,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 02:54:13,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRollbackAndDoubleExecution 2023-04-12 02:54:13,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:13,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:13,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:13,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testRollbackAndDoubleExecution' member 'jenkins-hbase4.apache.org,44679,1681268031772': 2023-04-12 02:54:13,576 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44679,1681268031772' released barrier for procedure'testRollbackAndDoubleExecution', counting down latch. Waiting for 0 more 2023-04-12 02:54:13,576 INFO [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testRollbackAndDoubleExecution' execution completed 2023-04-12 02:54:13,577 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 02:54:13,577 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 02:54:13,577 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testRollbackAndDoubleExecution 2023-04-12 02:54:13,577 INFO [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testRollbackAndDoubleExecutionincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 02:54:13,578 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-12 02:54:13,578 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-12 02:54:13,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-12 02:54:13,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 02:54:13,579 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-12 02:54:13,579 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-12 02:54:13,579 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-12 02:54:13,578 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:54:13,578 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-12 02:54:13,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-12 02:54:13,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-12 02:54:13,579 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:54:13,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 02:54:13,579 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:13,579 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-12 02:54:13,579 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:54:13,579 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:54:13,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:54:13,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-12 02:54:13,579 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:54:13,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:54:13,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 02:54:13,580 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:13,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-12 02:54:13,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-12 02:54:13,580 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:54:13,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:54:13,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRollbackAndDoubleExecution 2023-04-12 02:54:13,580 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:13,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-12 02:54:13,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 02:54:13,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRollbackAndDoubleExecution 2023-04-12 02:54:13,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:13,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:13,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:13,582 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:13,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 02:54:13,583 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:13,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRollbackAndDoubleExecution 2023-04-12 02:54:13,583 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:13,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:13,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:13,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:13,586 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:54:13,586 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:54:13,586 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:54:13,586 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:54:13,586 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:54:13,586 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:13,586 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:54:13,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:54:13,586 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:54:13,586 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:54:13,586 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:54:13,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:54:13,586 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:54:13,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:54:13,586 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testRollbackAndDoubleExecution' 2023-04-12 02:54:13,587 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 02:54:13,586 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-04-12 02:54:13,586 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:54:13,587 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:13,587 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:13,587 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-04-12 02:54:13,587 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:54:13,587 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:54:13,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:54:13,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:54:13,587 DEBUG [Listener at localhost/46667] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testRollbackAndDoubleExecution'' to complete. (max 20000 ms per retry) 2023-04-12 02:54:13,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:54:13,587 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:13,587 DEBUG [Listener at localhost/46667] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 02:54:13,587 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-12 02:54:13,587 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:13,587 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:13,588 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-12 02:54:13,588 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-12 02:54:13,687 DEBUG [Listener at localhost/46667] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 02:54:13,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 02:54:13,724 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 02:54:13,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] procedure.ProcedureCoordinator(143): Procedure testRollbackAndDoubleExecution was in running list but was completed. Accepting new attempt. 2023-04-12 02:54:13,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] procedure.ProcedureCoordinator(165): Submitting procedure testRollbackAndDoubleExecution 2023-04-12 02:54:13,727 INFO [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testRollbackAndDoubleExecution' 2023-04-12 02:54:13,727 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:54:13,727 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testRollbackAndDoubleExecution' starting 'acquire' 2023-04-12 02:54:13,727 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testRollbackAndDoubleExecution', kicking off acquire phase on members. 2023-04-12 02:54:13,728 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-12 02:54:13,728 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-04-12 02:54:13,731 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:54:13,731 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:54:13,731 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:54:13,731 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:54:13,731 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:13,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:54:13,731 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:54:13,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:54:13,731 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:54:13,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:54:13,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-04-12 02:54:13,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-04-12 02:54:13,732 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-12 02:54:13,732 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-12 02:54:13,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-04-12 02:54:13,732 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:13,732 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 02:54:13,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 02:54:13,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-04-12 02:54:13,733 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testRollbackAndDoubleExecution 2023-04-12 02:54:13,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 02:54:13,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-04-12 02:54:13,733 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testRollbackAndDoubleExecution 2023-04-12 02:54:13,733 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-12 02:54:13,733 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testRollbackAndDoubleExecution is still present, removing 2023-04-12 02:54:13,733 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testRollbackAndDoubleExecution 2023-04-12 02:54:13,733 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testRollbackAndDoubleExecution' with timeout 60000ms 2023-04-12 02:54:13,734 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:54:13,734 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testRollbackAndDoubleExecution' starting 'acquire' stage 2023-04-12 02:54:13,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 02:54:13,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-04-12 02:54:13,734 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testRollbackAndDoubleExecution 2023-04-12 02:54:13,734 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testRollbackAndDoubleExecution' locally acquired 2023-04-12 02:54:13,734 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43091,1681268031804' joining acquired barrier for procedure (testRollbackAndDoubleExecution) in zk 2023-04-12 02:54:13,734 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testRollbackAndDoubleExecution is still present, removing 2023-04-12 02:54:13,734 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testRollbackAndDoubleExecution 2023-04-12 02:54:13,734 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testRollbackAndDoubleExecution' with timeout 60000ms 2023-04-12 02:54:13,734 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:54:13,734 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testRollbackAndDoubleExecution' starting 'acquire' stage 2023-04-12 02:54:13,734 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testRollbackAndDoubleExecution' locally acquired 2023-04-12 02:54:13,735 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34579,1681268031719' joining acquired barrier for procedure (testRollbackAndDoubleExecution) in zk 2023-04-12 02:54:13,739 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-12 02:54:13,739 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-12 02:54:13,740 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testRollbackAndDoubleExecution' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 02:54:13,740 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testRollbackAndDoubleExecution is still present, removing 2023-04-12 02:54:13,740 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testRollbackAndDoubleExecution 2023-04-12 02:54:13,740 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testRollbackAndDoubleExecution' with timeout 60000ms 2023-04-12 02:54:13,740 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:54:13,740 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testRollbackAndDoubleExecution' starting 'acquire' stage 2023-04-12 02:54:13,740 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-12 02:54:13,741 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-12 02:54:13,741 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testRollbackAndDoubleExecution' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 02:54:13,748 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 02:54:13,748 DEBUG [rs(jenkins-hbase4.apache.org,44679,1681268031772)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testRollbackAndDoubleExecution,,1681268051553.96cdae2f7638212730b694522b9a464a. 2023-04-12 02:54:13,748 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 02:54:13,748 DEBUG [rs(jenkins-hbase4.apache.org,44679,1681268031772)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testRollbackAndDoubleExecution,,1681268051553.96cdae2f7638212730b694522b9a464a. started... 2023-04-12 02:54:13,748 INFO [rs(jenkins-hbase4.apache.org,44679,1681268031772)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 96cdae2f7638212730b694522b9a464a 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-04-12 02:54:13,807 INFO [rs(jenkins-hbase4.apache.org,44679,1681268031772)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=71 (bloomFilter=true), to=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/.tmp/cf1/b5b939a0ed784022a8f999fa3edea7b9 2023-04-12 02:54:13,867 INFO [rs(jenkins-hbase4.apache.org,44679,1681268031772)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=71 (bloomFilter=true), to=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/.tmp/cf2/68757554e4ad4422867a2933985cb078 2023-04-12 02:54:13,876 DEBUG [rs(jenkins-hbase4.apache.org,44679,1681268031772)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/.tmp/cf1/b5b939a0ed784022a8f999fa3edea7b9 as hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/cf1/b5b939a0ed784022a8f999fa3edea7b9 2023-04-12 02:54:13,884 INFO [rs(jenkins-hbase4.apache.org,44679,1681268031772)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/cf1/b5b939a0ed784022a8f999fa3edea7b9, entries=10, sequenceid=71, filesize=5.0 K 2023-04-12 02:54:13,885 DEBUG [rs(jenkins-hbase4.apache.org,44679,1681268031772)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/.tmp/cf2/68757554e4ad4422867a2933985cb078 as hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/cf2/68757554e4ad4422867a2933985cb078 2023-04-12 02:54:13,909 INFO [rs(jenkins-hbase4.apache.org,44679,1681268031772)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/cf2/68757554e4ad4422867a2933985cb078, entries=10, sequenceid=71, filesize=5.0 K 2023-04-12 02:54:13,911 INFO [rs(jenkins-hbase4.apache.org,44679,1681268031772)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for 96cdae2f7638212730b694522b9a464a in 163ms, sequenceid=71, compaction requested=true 2023-04-12 02:54:13,911 DEBUG [rs(jenkins-hbase4.apache.org,44679,1681268031772)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 96cdae2f7638212730b694522b9a464a: 2023-04-12 02:54:13,911 DEBUG [rs(jenkins-hbase4.apache.org,44679,1681268031772)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testRollbackAndDoubleExecution,,1681268051553.96cdae2f7638212730b694522b9a464a. 2023-04-12 02:54:13,911 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 02:54:13,911 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 02:54:13,911 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:13,911 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testRollbackAndDoubleExecution' locally acquired 2023-04-12 02:54:13,911 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44679,1681268031772' joining acquired barrier for procedure (testRollbackAndDoubleExecution) in zk 2023-04-12 02:54:13,913 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-12 02:54:13,913 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:13,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:13,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 02:54:13,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 02:54:13,914 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-12 02:54:13,914 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testRollbackAndDoubleExecution' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 02:54:13,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 02:54:13,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 02:54:13,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRollbackAndDoubleExecution 2023-04-12 02:54:13,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:13,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:13,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:13,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 02:54:13,918 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,44679,1681268031772' joining acquired barrier for procedure 'testRollbackAndDoubleExecution' on coordinator 2023-04-12 02:54:13,918 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@206689d0[Count = 0] remaining members to acquire global barrier 2023-04-12 02:54:13,918 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testRollbackAndDoubleExecution' starting 'in-barrier' execution. 2023-04-12 02:54:13,918 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-12 02:54:13,920 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-12 02:54:13,920 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-12 02:54:13,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-12 02:54:13,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-12 02:54:13,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-12 02:54:13,920 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-12 02:54:13,921 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testRollbackAndDoubleExecution' received 'reached' from coordinator. 2023-04-12 02:54:13,921 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testRollbackAndDoubleExecution' locally completed 2023-04-12 02:54:13,921 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testRollbackAndDoubleExecution' completed for member 'jenkins-hbase4.apache.org,43091,1681268031804' in zk 2023-04-12 02:54:13,921 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-12 02:54:13,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-12 02:54:13,921 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testRollbackAndDoubleExecution' received 'reached' from coordinator. 2023-04-12 02:54:13,921 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testRollbackAndDoubleExecution' locally completed 2023-04-12 02:54:13,921 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testRollbackAndDoubleExecution' completed for member 'jenkins-hbase4.apache.org,34579,1681268031719' in zk 2023-04-12 02:54:13,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-12 02:54:13,921 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testRollbackAndDoubleExecution' received 'reached' from coordinator. 2023-04-12 02:54:13,921 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testRollbackAndDoubleExecution' locally completed 2023-04-12 02:54:13,921 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testRollbackAndDoubleExecution' completed for member 'jenkins-hbase4.apache.org,44679,1681268031772' in zk 2023-04-12 02:54:13,923 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:13,923 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 02:54:13,924 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testRollbackAndDoubleExecution' has notified controller of completion 2023-04-12 02:54:13,924 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:54:13,924 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testRollbackAndDoubleExecution' has notified controller of completion 2023-04-12 02:54:13,924 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:54:13,924 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testRollbackAndDoubleExecution' completed. 2023-04-12 02:54:13,924 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testRollbackAndDoubleExecution' completed. 2023-04-12 02:54:13,926 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testRollbackAndDoubleExecution' has notified controller of completion 2023-04-12 02:54:13,926 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:54:13,926 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:13,926 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testRollbackAndDoubleExecution' completed. 2023-04-12 02:54:13,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:13,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 02:54:13,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 02:54:13,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 02:54:13,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 02:54:13,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRollbackAndDoubleExecution 2023-04-12 02:54:13,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:13,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:13,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:13,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 02:54:13,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRollbackAndDoubleExecution 2023-04-12 02:54:13,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:13,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:13,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:13,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testRollbackAndDoubleExecution' member 'jenkins-hbase4.apache.org,44679,1681268031772': 2023-04-12 02:54:13,932 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,44679,1681268031772' released barrier for procedure'testRollbackAndDoubleExecution', counting down latch. Waiting for 0 more 2023-04-12 02:54:13,932 INFO [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testRollbackAndDoubleExecution' execution completed 2023-04-12 02:54:13,933 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 02:54:13,933 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 02:54:13,933 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testRollbackAndDoubleExecution 2023-04-12 02:54:13,933 INFO [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testRollbackAndDoubleExecutionincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 02:54:13,935 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-12 02:54:13,935 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:54:13,935 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-12 02:54:13,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-12 02:54:13,935 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-12 02:54:13,936 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:54:13,936 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-12 02:54:13,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-12 02:54:13,936 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-12 02:54:13,936 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:54:13,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:54:13,936 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-12 02:54:13,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-12 02:54:13,936 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:54:13,936 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:54:13,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:54:13,937 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-12 02:54:13,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-12 02:54:13,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 02:54:13,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 02:54:13,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-12 02:54:13,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-12 02:54:13,938 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:54:13,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:54:13,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 02:54:13,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-12 02:54:13,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRollbackAndDoubleExecution 2023-04-12 02:54:13,939 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:13,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 02:54:13,940 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:13,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRollbackAndDoubleExecution 2023-04-12 02:54:13,941 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:13,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:13,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:13,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:13,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 02:54:13,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRollbackAndDoubleExecution 2023-04-12 02:54:13,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:13,944 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:13,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:13,945 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:13,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:13,949 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:13,959 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:54:13,959 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:54:13,959 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:54:13,959 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:54:13,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:54:13,959 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:54:13,959 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:13,959 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-04-12 02:54:13,959 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:13,959 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:13,959 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-04-12 02:54:13,959 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:54:13,960 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:54:13,960 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:54:13,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:54:13,960 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testRollbackAndDoubleExecution' 2023-04-12 02:54:13,960 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 02:54:13,960 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:54:13,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:54:13,959 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:13,960 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-12 02:54:13,960 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:13,960 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:13,960 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-12 02:54:13,960 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-12 02:54:13,959 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:54:13,960 DEBUG [Listener at localhost/46667] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testRollbackAndDoubleExecution'' to complete. (max 20000 ms per retry) 2023-04-12 02:54:13,960 DEBUG [Listener at localhost/46667] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 02:54:13,960 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:54:13,960 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:54:13,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:54:13,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:54:13,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:54:13,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:54:14,061 DEBUG [Listener at localhost/46667] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 02:54:14,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 02:54:14,341 WARN [Listener at localhost/46667] procedure2.ProcedureTestingUtility(193): Set Kill before store update to: true 2023-04-12 02:54:14,342 DEBUG [Listener at localhost/46667] assignment.SplitTableRegionProcedure(219): Splittable=true state=OPEN, location=jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:14,343 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(1029): Stored pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=96cdae2f7638212730b694522b9a464a, daughterA=fdd18e9cf539c13d863267ce16031583, daughterB=578095173fbc35e181cce14464037c2e 2023-04-12 02:54:14,345 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=96cdae2f7638212730b694522b9a464a, daughterA=fdd18e9cf539c13d863267ce16031583, daughterB=578095173fbc35e181cce14464037c2e 2023-04-12 02:54:14,345 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=96cdae2f7638212730b694522b9a464a, daughterA=fdd18e9cf539c13d863267ce16031583, daughterB=578095173fbc35e181cce14464037c2e 2023-04-12 02:54:14,345 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=96cdae2f7638212730b694522b9a464a, daughterA=fdd18e9cf539c13d863267ce16031583, daughterB=578095173fbc35e181cce14464037c2e 2023-04-12 02:54:14,346 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-04-12 02:54:14,346 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION, locked=true; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=96cdae2f7638212730b694522b9a464a, daughterA=fdd18e9cf539c13d863267ce16031583, daughterB=578095173fbc35e181cce14464037c2e 2023-04-12 02:54:14,346 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-04-12 02:54:14,346 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION, locked=true; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=96cdae2f7638212730b694522b9a464a, daughterA=fdd18e9cf539c13d863267ce16031583, daughterB=578095173fbc35e181cce14464037c2e at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-04-12 02:54:14,594 INFO [Listener at localhost/46667] procedure.MasterProcedureTestingUtility(417): Restart 2 exec state=pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION, locked=true; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=96cdae2f7638212730b694522b9a464a, daughterA=fdd18e9cf539c13d863267ce16031583, daughterB=578095173fbc35e181cce14464037c2e 2023-04-12 02:54:14,594 INFO [Listener at localhost/46667] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-04-12 02:54:14,594 INFO [Listener at localhost/46667] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-04-12 02:54:14,594 INFO [Listener at localhost/46667] assignment.AssignmentManager(315): Stopping assignment manager 2023-04-12 02:54:14,594 ERROR [Listener at localhost/46667] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] Thread[HFileArchiver-7,5,PEWorkerGroup] Thread[HFileArchiver-8,5,PEWorkerGroup] 2023-04-12 02:54:14,594 INFO [Listener at localhost/46667] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-04-12 02:54:14,594 INFO [Listener at localhost/46667] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-04-12 02:54:14,594 INFO [Listener at localhost/46667] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-04-12 02:54:14,594 INFO [Listener at localhost/46667] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-04-12 02:54:14,595 INFO [Listener at localhost/46667] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-04-12 02:54:14,597 DEBUG [Listener at localhost/46667] store.ProcedureTree(114): Procedure Procedure(pid=44, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[] 2023-04-12 02:54:14,597 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-04-12 02:54:14,597 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-04-12 02:54:14,598 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-04-12 02:54:14,598 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-04-12 02:54:14,598 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-12 02:54:14,598 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=cc9012ec0b656baa24d04368fbe026c4, daughterA=e2a9b8fe15a781798c0b49e48ff5aad7, daughterB=6f55b5d6a460bd8c7569dee22f049104 2023-04-12 02:54:14,599 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-12 02:54:14,599 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-12 02:54:14,599 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-04-12 02:54:14,599 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=28, state=SUCCESS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=315e13e868a7f68a3ce37802b2f6312d, daughterA=4e1092f3c00484f8f09c730861225d43, daughterB=291734c00d212fd203b04cf8872d4649 2023-04-12 02:54:14,600 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=35, state=SUCCESS; DisableTableProcedure table=testSplitWithoutPONR 2023-04-12 02:54:14,600 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=40, state=SUCCESS; DeleteTableProcedure table=testSplitWithoutPONR 2023-04-12 02:54:14,600 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=41, state=SUCCESS; CreateTableProcedure table=testRollbackAndDoubleExecution 2023-04-12 02:54:14,601 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(468): Loading pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=96cdae2f7638212730b694522b9a464a, daughterA=fdd18e9cf539c13d863267ce16031583, daughterB=578095173fbc35e181cce14464037c2e 2023-04-12 02:54:14,601 DEBUG [Listener at localhost/46667] procedure2.Procedure(966): pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=96cdae2f7638212730b694522b9a464a, daughterA=fdd18e9cf539c13d863267ce16031583, daughterB=578095173fbc35e181cce14464037c2e held the lock before restarting, call acquireLock to restore it. 2023-04-12 02:54:14,601 INFO [Listener at localhost/46667] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=96cdae2f7638212730b694522b9a464a, daughterA=fdd18e9cf539c13d863267ce16031583, daughterB=578095173fbc35e181cce14464037c2e 2023-04-12 02:54:14,601 INFO [Listener at localhost/46667] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=96cdae2f7638212730b694522b9a464a, daughterA=fdd18e9cf539c13d863267ce16031583, daughterB=578095173fbc35e181cce14464037c2e 2023-04-12 02:54:14,601 INFO [Listener at localhost/46667] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=96cdae2f7638212730b694522b9a464a, daughterA=fdd18e9cf539c13d863267ce16031583, daughterB=578095173fbc35e181cce14464037c2e 2023-04-12 02:54:14,601 INFO [Listener at localhost/46667] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 5 msec 2023-04-12 02:54:14,602 DEBUG [Listener at localhost/46667] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-04-12 02:54:14,602 DEBUG [Listener at localhost/46667] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase4.apache.org,44679,1681268031772, table=hbase:meta, region=1588230740 2023-04-12 02:54:14,604 INFO [Listener at localhost/46667] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1681268084604 2023-04-12 02:54:14,605 DEBUG [Listener at localhost/46667] assignment.AssignmentManager(1517): Joining cluster... 2023-04-12 02:54:14,605 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=96cdae2f7638212730b694522b9a464a, daughterA=fdd18e9cf539c13d863267ce16031583, daughterB=578095173fbc35e181cce14464037c2e 2023-04-12 02:54:14,608 DEBUG [Listener at localhost/46667] assignment.RegionStateStore(147): Load hbase:meta entry region=1f255e980a31bb266f2206a30aa2ae3f, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,44679,1681268031772, regionLocation=jenkins-hbase4.apache.org,44679,1681268031772, openSeqNum=2 2023-04-12 02:54:14,608 DEBUG [Listener at localhost/46667] assignment.RegionStateStore(147): Load hbase:meta entry region=96cdae2f7638212730b694522b9a464a, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,44679,1681268031772, regionLocation=jenkins-hbase4.apache.org,44679,1681268031772, openSeqNum=2 2023-04-12 02:54:14,608 INFO [Listener at localhost/46667] assignment.AssignmentManager(1529): Number of RegionServers=3 2023-04-12 02:54:14,608 INFO [Listener at localhost/46667] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1681268114608 2023-04-12 02:54:14,608 INFO [Listener at localhost/46667] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1681268174608 2023-04-12 02:54:14,608 INFO [Listener at localhost/46667] assignment.AssignmentManager(1536): Joined the cluster in 2 msec 2023-04-12 02:54:14,609 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-04-12 02:54:14,611 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-04-12 02:54:14,611 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_CLOSE_PARENT_REGION, locked=true; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=96cdae2f7638212730b694522b9a464a, daughterA=fdd18e9cf539c13d863267ce16031583, daughterB=578095173fbc35e181cce14464037c2e 2023-04-12 02:54:14,611 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-04-12 02:54:14,611 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_CLOSE_PARENT_REGION, locked=true; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=96cdae2f7638212730b694522b9a464a, daughterA=fdd18e9cf539c13d863267ce16031583, daughterB=578095173fbc35e181cce14464037c2e at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-04-12 02:54:14,859 INFO [Listener at localhost/46667] procedure.MasterProcedureTestingUtility(417): Restart 3 exec state=pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_CLOSE_PARENT_REGION, locked=true; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=96cdae2f7638212730b694522b9a464a, daughterA=fdd18e9cf539c13d863267ce16031583, daughterB=578095173fbc35e181cce14464037c2e 2023-04-12 02:54:14,859 INFO [Listener at localhost/46667] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-04-12 02:54:14,859 INFO [Listener at localhost/46667] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-04-12 02:54:14,859 INFO [Listener at localhost/46667] assignment.AssignmentManager(315): Stopping assignment manager 2023-04-12 02:54:14,860 ERROR [Listener at localhost/46667] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-04-12 02:54:14,860 INFO [Listener at localhost/46667] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-04-12 02:54:14,860 INFO [Listener at localhost/46667] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-04-12 02:54:14,860 INFO [Listener at localhost/46667] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-04-12 02:54:14,861 INFO [Listener at localhost/46667] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-04-12 02:54:14,861 INFO [Listener at localhost/46667] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-04-12 02:54:14,862 DEBUG [Listener at localhost/46667] store.ProcedureTree(114): Procedure Procedure(pid=44, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[0] 2023-04-12 02:54:14,863 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-04-12 02:54:14,863 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-04-12 02:54:14,863 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-04-12 02:54:14,864 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-04-12 02:54:14,864 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-12 02:54:14,864 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=cc9012ec0b656baa24d04368fbe026c4, daughterA=e2a9b8fe15a781798c0b49e48ff5aad7, daughterB=6f55b5d6a460bd8c7569dee22f049104 2023-04-12 02:54:14,865 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-12 02:54:14,865 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-12 02:54:14,865 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-04-12 02:54:14,866 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=28, state=SUCCESS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=315e13e868a7f68a3ce37802b2f6312d, daughterA=4e1092f3c00484f8f09c730861225d43, daughterB=291734c00d212fd203b04cf8872d4649 2023-04-12 02:54:14,866 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=35, state=SUCCESS; DisableTableProcedure table=testSplitWithoutPONR 2023-04-12 02:54:14,866 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=40, state=SUCCESS; DeleteTableProcedure table=testSplitWithoutPONR 2023-04-12 02:54:14,867 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=41, state=SUCCESS; CreateTableProcedure table=testRollbackAndDoubleExecution 2023-04-12 02:54:14,867 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(468): Loading pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=96cdae2f7638212730b694522b9a464a, daughterA=fdd18e9cf539c13d863267ce16031583, daughterB=578095173fbc35e181cce14464037c2e 2023-04-12 02:54:14,867 DEBUG [Listener at localhost/46667] procedure2.Procedure(966): pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=96cdae2f7638212730b694522b9a464a, daughterA=fdd18e9cf539c13d863267ce16031583, daughterB=578095173fbc35e181cce14464037c2e held the lock before restarting, call acquireLock to restore it. 2023-04-12 02:54:14,867 INFO [Listener at localhost/46667] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=96cdae2f7638212730b694522b9a464a, daughterA=fdd18e9cf539c13d863267ce16031583, daughterB=578095173fbc35e181cce14464037c2e 2023-04-12 02:54:14,867 INFO [Listener at localhost/46667] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=96cdae2f7638212730b694522b9a464a, daughterA=fdd18e9cf539c13d863267ce16031583, daughterB=578095173fbc35e181cce14464037c2e 2023-04-12 02:54:14,867 INFO [Listener at localhost/46667] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=96cdae2f7638212730b694522b9a464a, daughterA=fdd18e9cf539c13d863267ce16031583, daughterB=578095173fbc35e181cce14464037c2e 2023-04-12 02:54:14,867 INFO [Listener at localhost/46667] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 5 msec 2023-04-12 02:54:14,879 DEBUG [Listener at localhost/46667] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-04-12 02:54:14,879 DEBUG [Listener at localhost/46667] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase4.apache.org,44679,1681268031772, table=hbase:meta, region=1588230740 2023-04-12 02:54:14,881 INFO [Listener at localhost/46667] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1681268084881 2023-04-12 02:54:14,883 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=96cdae2f7638212730b694522b9a464a, daughterA=fdd18e9cf539c13d863267ce16031583, daughterB=578095173fbc35e181cce14464037c2e 2023-04-12 02:54:14,883 DEBUG [Listener at localhost/46667] assignment.AssignmentManager(1517): Joining cluster... 2023-04-12 02:54:14,886 DEBUG [Listener at localhost/46667] assignment.RegionStateStore(147): Load hbase:meta entry region=1f255e980a31bb266f2206a30aa2ae3f, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,44679,1681268031772, regionLocation=jenkins-hbase4.apache.org,44679,1681268031772, openSeqNum=2 2023-04-12 02:54:14,886 DEBUG [Listener at localhost/46667] assignment.RegionStateStore(147): Load hbase:meta entry region=96cdae2f7638212730b694522b9a464a, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,44679,1681268031772, regionLocation=jenkins-hbase4.apache.org,44679,1681268031772, openSeqNum=2 2023-04-12 02:54:14,887 INFO [Listener at localhost/46667] assignment.AssignmentManager(1529): Number of RegionServers=3 2023-04-12 02:54:14,887 INFO [Listener at localhost/46667] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1681268114887 2023-04-12 02:54:14,887 INFO [Listener at localhost/46667] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1681268174887 2023-04-12 02:54:14,887 INFO [Listener at localhost/46667] assignment.AssignmentManager(1536): Joined the cluster in 3 msec 2023-04-12 02:54:14,887 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-04-12 02:54:14,889 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=45, ppid=44, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=96cdae2f7638212730b694522b9a464a, UNASSIGN}] 2023-04-12 02:54:14,889 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-04-12 02:54:14,889 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=44, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS, locked=true; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=96cdae2f7638212730b694522b9a464a, daughterA=fdd18e9cf539c13d863267ce16031583, daughterB=578095173fbc35e181cce14464037c2e 2023-04-12 02:54:14,889 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-04-12 02:54:14,889 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=44, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS, locked=true; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=96cdae2f7638212730b694522b9a464a, daughterA=fdd18e9cf539c13d863267ce16031583, daughterB=578095173fbc35e181cce14464037c2e at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-04-12 02:54:15,137 INFO [Listener at localhost/46667] procedure.MasterProcedureTestingUtility(417): Restart 11 exec state=pid=44, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS, locked=true; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=96cdae2f7638212730b694522b9a464a, daughterA=fdd18e9cf539c13d863267ce16031583, daughterB=578095173fbc35e181cce14464037c2e 2023-04-12 02:54:15,137 INFO [Listener at localhost/46667] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-04-12 02:54:15,137 INFO [Listener at localhost/46667] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-04-12 02:54:15,137 INFO [Listener at localhost/46667] assignment.AssignmentManager(315): Stopping assignment manager 2023-04-12 02:54:15,141 ERROR [Listener at localhost/46667] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-04-12 02:54:15,141 INFO [Listener at localhost/46667] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-04-12 02:54:15,141 INFO [Listener at localhost/46667] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-04-12 02:54:15,141 INFO [Listener at localhost/46667] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-04-12 02:54:15,141 INFO [Listener at localhost/46667] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-04-12 02:54:15,142 INFO [Listener at localhost/46667] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-04-12 02:54:15,143 DEBUG [Listener at localhost/46667] store.ProcedureTree(114): Procedure Procedure(pid=44, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[0, 1] 2023-04-12 02:54:15,151 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-04-12 02:54:15,152 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-04-12 02:54:15,152 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-04-12 02:54:15,152 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-04-12 02:54:15,152 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-12 02:54:15,153 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=cc9012ec0b656baa24d04368fbe026c4, daughterA=e2a9b8fe15a781798c0b49e48ff5aad7, daughterB=6f55b5d6a460bd8c7569dee22f049104 2023-04-12 02:54:15,153 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-12 02:54:15,153 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-12 02:54:15,154 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-04-12 02:54:15,154 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=28, state=SUCCESS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=315e13e868a7f68a3ce37802b2f6312d, daughterA=4e1092f3c00484f8f09c730861225d43, daughterB=291734c00d212fd203b04cf8872d4649 2023-04-12 02:54:15,155 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=35, state=SUCCESS; DisableTableProcedure table=testSplitWithoutPONR 2023-04-12 02:54:15,156 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=40, state=SUCCESS; DeleteTableProcedure table=testSplitWithoutPONR 2023-04-12 02:54:15,156 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=41, state=SUCCESS; CreateTableProcedure table=testRollbackAndDoubleExecution 2023-04-12 02:54:15,156 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(468): Loading pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_CLOSE_PARENT_REGION; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=96cdae2f7638212730b694522b9a464a, daughterA=fdd18e9cf539c13d863267ce16031583, daughterB=578095173fbc35e181cce14464037c2e 2023-04-12 02:54:15,157 DEBUG [Listener at localhost/46667] procedure2.Procedure(966): pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_CLOSE_PARENT_REGION; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=96cdae2f7638212730b694522b9a464a, daughterA=fdd18e9cf539c13d863267ce16031583, daughterB=578095173fbc35e181cce14464037c2e held the lock before restarting, call acquireLock to restore it. 2023-04-12 02:54:15,157 INFO [Listener at localhost/46667] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_CLOSE_PARENT_REGION; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=96cdae2f7638212730b694522b9a464a, daughterA=fdd18e9cf539c13d863267ce16031583, daughterB=578095173fbc35e181cce14464037c2e 2023-04-12 02:54:15,157 INFO [Listener at localhost/46667] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_CLOSE_PARENT_REGION; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=96cdae2f7638212730b694522b9a464a, daughterA=fdd18e9cf539c13d863267ce16031583, daughterB=578095173fbc35e181cce14464037c2e 2023-04-12 02:54:15,157 INFO [Listener at localhost/46667] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_CLOSE_PARENT_REGION; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=96cdae2f7638212730b694522b9a464a, daughterA=fdd18e9cf539c13d863267ce16031583, daughterB=578095173fbc35e181cce14464037c2e 2023-04-12 02:54:15,157 INFO [Listener at localhost/46667] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 14 msec 2023-04-12 02:54:15,162 DEBUG [Listener at localhost/46667] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-04-12 02:54:15,163 DEBUG [Listener at localhost/46667] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase4.apache.org,44679,1681268031772, table=hbase:meta, region=1588230740 2023-04-12 02:54:15,163 INFO [Listener at localhost/46667] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1681268085163 2023-04-12 02:54:15,164 DEBUG [Listener at localhost/46667] assignment.AssignmentManager(1517): Joining cluster... 2023-04-12 02:54:15,165 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_CLOSE_PARENT_REGION; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=96cdae2f7638212730b694522b9a464a, daughterA=fdd18e9cf539c13d863267ce16031583, daughterB=578095173fbc35e181cce14464037c2e 2023-04-12 02:54:15,168 DEBUG [Listener at localhost/46667] assignment.RegionStateStore(147): Load hbase:meta entry region=1f255e980a31bb266f2206a30aa2ae3f, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,44679,1681268031772, regionLocation=jenkins-hbase4.apache.org,44679,1681268031772, openSeqNum=2 2023-04-12 02:54:15,168 DEBUG [Listener at localhost/46667] assignment.RegionStateStore(147): Load hbase:meta entry region=96cdae2f7638212730b694522b9a464a, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,44679,1681268031772, regionLocation=jenkins-hbase4.apache.org,44679,1681268031772, openSeqNum=2 2023-04-12 02:54:15,168 INFO [Listener at localhost/46667] assignment.AssignmentManager(1529): Number of RegionServers=3 2023-04-12 02:54:15,168 INFO [Listener at localhost/46667] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1681268115168 2023-04-12 02:54:15,168 INFO [Listener at localhost/46667] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1681268175168 2023-04-12 02:54:15,168 INFO [Listener at localhost/46667] assignment.AssignmentManager(1536): Joined the cluster in 4 msec 2023-04-12 02:54:15,169 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=45, ppid=44, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=96cdae2f7638212730b694522b9a464a, UNASSIGN}] 2023-04-12 02:54:15,169 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-04-12 02:54:15,173 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=45, ppid=44, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=96cdae2f7638212730b694522b9a464a, UNASSIGN 2023-04-12 02:54:15,174 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=45 updating hbase:meta row=96cdae2f7638212730b694522b9a464a, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:15,174 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRollbackAndDoubleExecution,,1681268051553.96cdae2f7638212730b694522b9a464a.","families":{"info":[{"qualifier":"regioninfo","vlen":64,"tag":[],"timestamp":"1681268055174"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681268055174"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681268055174"}]},"ts":"1681268055174"} 2023-04-12 02:54:15,178 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=46, ppid=45, state=RUNNABLE; CloseRegionProcedure 96cdae2f7638212730b694522b9a464a, server=jenkins-hbase4.apache.org,44679,1681268031772}] 2023-04-12 02:54:15,178 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-04-12 02:54:15,178 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=45, ppid=44, state=WAITING:REGION_STATE_TRANSITION_CONFIRM_CLOSED, locked=true; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=96cdae2f7638212730b694522b9a464a, UNASSIGN 2023-04-12 02:54:15,178 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-04-12 02:54:15,178 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=45, ppid=44, state=WAITING:REGION_STATE_TRANSITION_CONFIRM_CLOSED, locked=true; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=96cdae2f7638212730b694522b9a464a, UNASSIGN at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-04-12 02:54:15,419 INFO [Listener at localhost/46667] procedure.MasterProcedureTestingUtility(417): Restart 11 exec state=pid=44, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS, locked=true; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=96cdae2f7638212730b694522b9a464a, daughterA=fdd18e9cf539c13d863267ce16031583, daughterB=578095173fbc35e181cce14464037c2e 2023-04-12 02:54:15,419 INFO [Listener at localhost/46667] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-04-12 02:54:15,419 INFO [Listener at localhost/46667] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-04-12 02:54:15,419 INFO [Listener at localhost/46667] assignment.AssignmentManager(315): Stopping assignment manager 2023-04-12 02:54:15,421 ERROR [Listener at localhost/46667] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-04-12 02:54:15,421 INFO [Listener at localhost/46667] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-04-12 02:54:15,421 INFO [Listener at localhost/46667] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-04-12 02:54:15,421 INFO [Listener at localhost/46667] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-04-12 02:54:15,421 INFO [Listener at localhost/46667] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-04-12 02:54:15,422 INFO [Listener at localhost/46667] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-04-12 02:54:15,423 DEBUG [Listener at localhost/46667] store.ProcedureTree(114): Procedure Procedure(pid=44, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[0, 1, 2] 2023-04-12 02:54:15,424 DEBUG [Listener at localhost/46667] store.ProcedureTree(114): Procedure Procedure(pid=45, ppid=44, class=org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure) stack ids=[] 2023-04-12 02:54:15,424 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-04-12 02:54:15,424 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-04-12 02:54:15,424 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-04-12 02:54:15,425 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-04-12 02:54:15,425 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-12 02:54:15,427 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=cc9012ec0b656baa24d04368fbe026c4, daughterA=e2a9b8fe15a781798c0b49e48ff5aad7, daughterB=6f55b5d6a460bd8c7569dee22f049104 2023-04-12 02:54:15,428 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-12 02:54:15,428 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-12 02:54:15,428 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-04-12 02:54:15,429 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=28, state=SUCCESS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=315e13e868a7f68a3ce37802b2f6312d, daughterA=4e1092f3c00484f8f09c730861225d43, daughterB=291734c00d212fd203b04cf8872d4649 2023-04-12 02:54:15,429 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=35, state=SUCCESS; DisableTableProcedure table=testSplitWithoutPONR 2023-04-12 02:54:15,429 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=40, state=SUCCESS; DeleteTableProcedure table=testSplitWithoutPONR 2023-04-12 02:54:15,429 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=41, state=SUCCESS; CreateTableProcedure table=testRollbackAndDoubleExecution 2023-04-12 02:54:15,430 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(468): Loading pid=44, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=96cdae2f7638212730b694522b9a464a, daughterA=fdd18e9cf539c13d863267ce16031583, daughterB=578095173fbc35e181cce14464037c2e 2023-04-12 02:54:15,430 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(468): Loading pid=45, ppid=44, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=96cdae2f7638212730b694522b9a464a, UNASSIGN 2023-04-12 02:54:15,430 DEBUG [Listener at localhost/46667] procedure2.Procedure(966): pid=44, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=96cdae2f7638212730b694522b9a464a, daughterA=fdd18e9cf539c13d863267ce16031583, daughterB=578095173fbc35e181cce14464037c2e held the lock before restarting, call acquireLock to restore it. 2023-04-12 02:54:15,430 INFO [Listener at localhost/46667] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=96cdae2f7638212730b694522b9a464a, daughterA=fdd18e9cf539c13d863267ce16031583, daughterB=578095173fbc35e181cce14464037c2e 2023-04-12 02:54:15,430 INFO [Listener at localhost/46667] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=96cdae2f7638212730b694522b9a464a, daughterA=fdd18e9cf539c13d863267ce16031583, daughterB=578095173fbc35e181cce14464037c2e 2023-04-12 02:54:15,430 INFO [Listener at localhost/46667] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=96cdae2f7638212730b694522b9a464a, daughterA=fdd18e9cf539c13d863267ce16031583, daughterB=578095173fbc35e181cce14464037c2e 2023-04-12 02:54:15,430 DEBUG [Listener at localhost/46667] procedure2.Procedure(966): pid=45, ppid=44, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=96cdae2f7638212730b694522b9a464a, UNASSIGN held the lock before restarting, call acquireLock to restore it. 2023-04-12 02:54:15,430 INFO [Listener at localhost/46667] procedure.MasterProcedureScheduler(727): Took xlock for pid=45, ppid=44, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=96cdae2f7638212730b694522b9a464a, UNASSIGN 2023-04-12 02:54:15,431 INFO [Listener at localhost/46667] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 8 msec 2023-04-12 02:54:15,437 DEBUG [Listener at localhost/46667] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-04-12 02:54:15,437 DEBUG [Listener at localhost/46667] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase4.apache.org,44679,1681268031772, table=hbase:meta, region=1588230740 2023-04-12 02:54:15,438 INFO [Listener at localhost/46667] assignment.AssignmentManager(305): Attach pid=45, ppid=44, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=96cdae2f7638212730b694522b9a464a, UNASSIGN to state=OFFLINE, location=null, table=testRollbackAndDoubleExecution, region=96cdae2f7638212730b694522b9a464a to restore RIT 2023-04-12 02:54:15,444 INFO [Listener at localhost/46667] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1681268085444 2023-04-12 02:54:15,445 DEBUG [Listener at localhost/46667] assignment.AssignmentManager(1517): Joining cluster... 2023-04-12 02:54:15,445 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=45, ppid=44, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=96cdae2f7638212730b694522b9a464a, UNASSIGN 2023-04-12 02:54:15,449 DEBUG [Listener at localhost/46667] assignment.RegionStateStore(147): Load hbase:meta entry region=1f255e980a31bb266f2206a30aa2ae3f, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,44679,1681268031772, regionLocation=jenkins-hbase4.apache.org,44679,1681268031772, openSeqNum=2 2023-04-12 02:54:15,449 DEBUG [Listener at localhost/46667] assignment.RegionStateStore(147): Load hbase:meta entry region=96cdae2f7638212730b694522b9a464a, regionState=CLOSING, lastHost=jenkins-hbase4.apache.org,44679,1681268031772, regionLocation=jenkins-hbase4.apache.org,44679,1681268031772, openSeqNum=2 2023-04-12 02:54:15,449 INFO [Listener at localhost/46667] assignment.AssignmentManager(1529): Number of RegionServers=3 2023-04-12 02:54:15,449 INFO [Listener at localhost/46667] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1681268115449 2023-04-12 02:54:15,449 INFO [Listener at localhost/46667] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1681268175449 2023-04-12 02:54:15,449 INFO [Listener at localhost/46667] assignment.AssignmentManager(1536): Joined the cluster in 4 msec 2023-04-12 02:54:15,450 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=45 updating hbase:meta row=96cdae2f7638212730b694522b9a464a, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:15,450 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRollbackAndDoubleExecution,,1681268051553.96cdae2f7638212730b694522b9a464a.","families":{"info":[{"qualifier":"regioninfo","vlen":64,"tag":[],"timestamp":"1681268055450"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681268055450"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681268055450"}]},"ts":"1681268055450"} 2023-04-12 02:54:15,452 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=46, ppid=45, state=RUNNABLE; CloseRegionProcedure 96cdae2f7638212730b694522b9a464a, server=jenkins-hbase4.apache.org,44679,1681268031772}] 2023-04-12 02:54:15,452 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-04-12 02:54:15,607 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 96cdae2f7638212730b694522b9a464a 2023-04-12 02:54:15,607 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 96cdae2f7638212730b694522b9a464a, disabling compactions & flushes 2023-04-12 02:54:15,607 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region testRollbackAndDoubleExecution,,1681268051553.96cdae2f7638212730b694522b9a464a. 2023-04-12 02:54:15,607 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testRollbackAndDoubleExecution,,1681268051553.96cdae2f7638212730b694522b9a464a. 2023-04-12 02:54:15,607 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on testRollbackAndDoubleExecution,,1681268051553.96cdae2f7638212730b694522b9a464a. after waiting 0 ms 2023-04-12 02:54:15,607 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region testRollbackAndDoubleExecution,,1681268051553.96cdae2f7638212730b694522b9a464a. 2023-04-12 02:54:15,607 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 96cdae2f7638212730b694522b9a464a 2/2 column families, dataSize=480 B heapSize=2.13 KB 2023-04-12 02:54:15,618 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=240 B at sequenceid=82 (bloomFilter=true), to=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/.tmp/cf1/a3b5b9d8fa954923ae97ea97984ade02 2023-04-12 02:54:15,631 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=240 B at sequenceid=82 (bloomFilter=true), to=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/.tmp/cf2/a0a6721d34874582969e0d913c5a3410 2023-04-12 02:54:15,635 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/.tmp/cf1/a3b5b9d8fa954923ae97ea97984ade02 as hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/cf1/a3b5b9d8fa954923ae97ea97984ade02 2023-04-12 02:54:15,640 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/cf1/a3b5b9d8fa954923ae97ea97984ade02, entries=8, sequenceid=82, filesize=5.0 K 2023-04-12 02:54:15,641 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/.tmp/cf2/a0a6721d34874582969e0d913c5a3410 as hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/cf2/a0a6721d34874582969e0d913c5a3410 2023-04-12 02:54:15,645 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/cf2/a0a6721d34874582969e0d913c5a3410, entries=8, sequenceid=82, filesize=5.0 K 2023-04-12 02:54:15,646 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~480 B/480, heapSize ~2.09 KB/2144, currentSize=0 B/0 for 96cdae2f7638212730b694522b9a464a in 39ms, sequenceid=82, compaction requested=true 2023-04-12 02:54:15,652 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/recovered.edits/85.seqid, newMaxSeqId=85, maxSeqId=1 2023-04-12 02:54:15,653 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro 2023-04-12 02:54:15,653 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed testRollbackAndDoubleExecution,,1681268051553.96cdae2f7638212730b694522b9a464a. 2023-04-12 02:54:15,653 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 96cdae2f7638212730b694522b9a464a: 2023-04-12 02:54:15,656 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 96cdae2f7638212730b694522b9a464a 2023-04-12 02:54:15,656 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=45 updating hbase:meta row=96cdae2f7638212730b694522b9a464a, regionState=CLOSED 2023-04-12 02:54:15,657 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRollbackAndDoubleExecution,,1681268051553.96cdae2f7638212730b694522b9a464a.","families":{"info":[{"qualifier":"regioninfo","vlen":64,"tag":[],"timestamp":"1681268055656"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681268055656"}]},"ts":"1681268055656"} 2023-04-12 02:54:15,658 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-04-12 02:54:15,658 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=46, ppid=45, state=SUCCESS, locked=true; CloseRegionProcedure 96cdae2f7638212730b694522b9a464a, server=jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:15,658 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-04-12 02:54:15,658 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=46, ppid=45, state=SUCCESS, locked=true; CloseRegionProcedure 96cdae2f7638212730b694522b9a464a, server=jenkins-hbase4.apache.org,44679,1681268031772 at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-04-12 02:54:15,700 INFO [Listener at localhost/46667] procedure.MasterProcedureTestingUtility(417): Restart 11 exec state=pid=44, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=96cdae2f7638212730b694522b9a464a, daughterA=fdd18e9cf539c13d863267ce16031583, daughterB=578095173fbc35e181cce14464037c2e 2023-04-12 02:54:15,700 INFO [Listener at localhost/46667] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-04-12 02:54:15,700 INFO [Listener at localhost/46667] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-04-12 02:54:15,700 INFO [Listener at localhost/46667] assignment.AssignmentManager(315): Stopping assignment manager 2023-04-12 02:54:15,701 ERROR [Listener at localhost/46667] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-04-12 02:54:15,701 INFO [Listener at localhost/46667] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-04-12 02:54:15,702 INFO [Listener at localhost/46667] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-04-12 02:54:15,702 INFO [Listener at localhost/46667] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-04-12 02:54:15,702 INFO [Listener at localhost/46667] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-04-12 02:54:15,702 INFO [Listener at localhost/46667] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-04-12 02:54:15,704 DEBUG [Listener at localhost/46667] store.ProcedureTree(114): Procedure Procedure(pid=44, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[0, 1, 2] 2023-04-12 02:54:15,704 DEBUG [Listener at localhost/46667] store.ProcedureTree(114): Procedure Procedure(pid=45, ppid=44, class=org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure) stack ids=[3] 2023-04-12 02:54:15,704 DEBUG [Listener at localhost/46667] store.ProcedureTree(114): Procedure Procedure(pid=46, ppid=45, class=org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure) stack ids=[4] 2023-04-12 02:54:15,704 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-04-12 02:54:15,705 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-04-12 02:54:15,705 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-04-12 02:54:15,705 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-04-12 02:54:15,706 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-12 02:54:15,706 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=cc9012ec0b656baa24d04368fbe026c4, daughterA=e2a9b8fe15a781798c0b49e48ff5aad7, daughterB=6f55b5d6a460bd8c7569dee22f049104 2023-04-12 02:54:15,706 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-12 02:54:15,707 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-12 02:54:15,707 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-04-12 02:54:15,707 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=28, state=SUCCESS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=315e13e868a7f68a3ce37802b2f6312d, daughterA=4e1092f3c00484f8f09c730861225d43, daughterB=291734c00d212fd203b04cf8872d4649 2023-04-12 02:54:15,708 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=35, state=SUCCESS; DisableTableProcedure table=testSplitWithoutPONR 2023-04-12 02:54:15,708 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=40, state=SUCCESS; DeleteTableProcedure table=testSplitWithoutPONR 2023-04-12 02:54:15,708 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=41, state=SUCCESS; CreateTableProcedure table=testRollbackAndDoubleExecution 2023-04-12 02:54:15,709 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(468): Loading pid=44, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=96cdae2f7638212730b694522b9a464a, daughterA=fdd18e9cf539c13d863267ce16031583, daughterB=578095173fbc35e181cce14464037c2e 2023-04-12 02:54:15,709 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(468): Loading pid=45, ppid=44, state=WAITING:REGION_STATE_TRANSITION_CONFIRM_CLOSED; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=96cdae2f7638212730b694522b9a464a, UNASSIGN 2023-04-12 02:54:15,709 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(468): Loading pid=46, ppid=45, state=RUNNABLE; CloseRegionProcedure 96cdae2f7638212730b694522b9a464a, server=jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:15,710 DEBUG [Listener at localhost/46667] procedure2.Procedure(966): pid=44, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=96cdae2f7638212730b694522b9a464a, daughterA=fdd18e9cf539c13d863267ce16031583, daughterB=578095173fbc35e181cce14464037c2e held the lock before restarting, call acquireLock to restore it. 2023-04-12 02:54:15,710 INFO [Listener at localhost/46667] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=96cdae2f7638212730b694522b9a464a, daughterA=fdd18e9cf539c13d863267ce16031583, daughterB=578095173fbc35e181cce14464037c2e 2023-04-12 02:54:15,710 INFO [Listener at localhost/46667] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=96cdae2f7638212730b694522b9a464a, daughterA=fdd18e9cf539c13d863267ce16031583, daughterB=578095173fbc35e181cce14464037c2e 2023-04-12 02:54:15,710 INFO [Listener at localhost/46667] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=96cdae2f7638212730b694522b9a464a, daughterA=fdd18e9cf539c13d863267ce16031583, daughterB=578095173fbc35e181cce14464037c2e 2023-04-12 02:54:15,710 DEBUG [Listener at localhost/46667] procedure2.Procedure(966): pid=45, ppid=44, state=WAITING:REGION_STATE_TRANSITION_CONFIRM_CLOSED; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=96cdae2f7638212730b694522b9a464a, UNASSIGN held the lock before restarting, call acquireLock to restore it. 2023-04-12 02:54:15,710 INFO [Listener at localhost/46667] procedure.MasterProcedureScheduler(727): Took xlock for pid=45, ppid=44, state=WAITING:REGION_STATE_TRANSITION_CONFIRM_CLOSED; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=96cdae2f7638212730b694522b9a464a, UNASSIGN 2023-04-12 02:54:15,711 DEBUG [Listener at localhost/46667] procedure2.Procedure(966): pid=46, ppid=45, state=RUNNABLE; CloseRegionProcedure 96cdae2f7638212730b694522b9a464a, server=jenkins-hbase4.apache.org,44679,1681268031772 held the lock before restarting, call acquireLock to restore it. 2023-04-12 02:54:15,711 INFO [Listener at localhost/46667] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 8 msec 2023-04-12 02:54:15,712 DEBUG [Listener at localhost/46667] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-04-12 02:54:15,712 DEBUG [Listener at localhost/46667] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase4.apache.org,44679,1681268031772, table=hbase:meta, region=1588230740 2023-04-12 02:54:15,713 INFO [Listener at localhost/46667] assignment.AssignmentManager(305): Attach pid=45, ppid=44, state=WAITING:REGION_STATE_TRANSITION_CONFIRM_CLOSED; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=96cdae2f7638212730b694522b9a464a, UNASSIGN to state=OFFLINE, location=null, table=testRollbackAndDoubleExecution, region=96cdae2f7638212730b694522b9a464a to restore RIT 2023-04-12 02:54:15,720 INFO [Listener at localhost/46667] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1681268085720 2023-04-12 02:54:15,721 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=46, ppid=45, state=RUNNABLE; CloseRegionProcedure 96cdae2f7638212730b694522b9a464a, server=jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:15,721 DEBUG [Listener at localhost/46667] assignment.AssignmentManager(1517): Joining cluster... 2023-04-12 02:54:15,724 DEBUG [Listener at localhost/46667] assignment.RegionStateStore(147): Load hbase:meta entry region=1f255e980a31bb266f2206a30aa2ae3f, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,44679,1681268031772, regionLocation=jenkins-hbase4.apache.org,44679,1681268031772, openSeqNum=2 2023-04-12 02:54:15,725 DEBUG [Listener at localhost/46667] assignment.RegionStateStore(147): Load hbase:meta entry region=96cdae2f7638212730b694522b9a464a, regionState=CLOSED, lastHost=jenkins-hbase4.apache.org,44679,1681268031772, regionLocation=jenkins-hbase4.apache.org,44679,1681268031772, openSeqNum=2 2023-04-12 02:54:15,725 INFO [Listener at localhost/46667] assignment.AssignmentManager(1529): Number of RegionServers=3 2023-04-12 02:54:15,725 INFO [Listener at localhost/46667] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1681268115725 2023-04-12 02:54:15,725 INFO [Listener at localhost/46667] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1681268175725 2023-04-12 02:54:15,725 INFO [Listener at localhost/46667] assignment.AssignmentManager(1536): Joined the cluster in 3 msec 2023-04-12 02:54:15,726 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=45 updating hbase:meta row=96cdae2f7638212730b694522b9a464a, regionState=CLOSED, regionLocation=jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:15,726 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRollbackAndDoubleExecution,,1681268051553.96cdae2f7638212730b694522b9a464a.","families":{"info":[{"qualifier":"regioninfo","vlen":64,"tag":[],"timestamp":"1681268055725"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681268055725"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681268055725"}]},"ts":"1681268055725"} 2023-04-12 02:54:15,737 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-04-12 02:54:15,748 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=46, resume processing ppid=45 2023-04-12 02:54:15,748 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=46, ppid=45, state=SUCCESS; CloseRegionProcedure 96cdae2f7638212730b694522b9a464a, server=jenkins-hbase4.apache.org,44679,1681268031772 in 285 msec 2023-04-12 02:54:15,748 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-04-12 02:54:15,748 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=45, ppid=44, state=SUCCESS, locked=true; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=96cdae2f7638212730b694522b9a464a, UNASSIGN 2023-04-12 02:54:15,748 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-04-12 02:54:15,748 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=45, ppid=44, state=SUCCESS, locked=true; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=96cdae2f7638212730b694522b9a464a, UNASSIGN at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-04-12 02:54:15,976 INFO [Listener at localhost/46667] procedure.MasterProcedureTestingUtility(417): Restart 11 exec state=pid=44, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=96cdae2f7638212730b694522b9a464a, daughterA=fdd18e9cf539c13d863267ce16031583, daughterB=578095173fbc35e181cce14464037c2e 2023-04-12 02:54:15,976 INFO [Listener at localhost/46667] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-04-12 02:54:15,976 INFO [Listener at localhost/46667] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-04-12 02:54:15,976 INFO [Listener at localhost/46667] assignment.AssignmentManager(315): Stopping assignment manager 2023-04-12 02:54:15,977 ERROR [Listener at localhost/46667] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-04-12 02:54:15,977 INFO [Listener at localhost/46667] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-04-12 02:54:15,978 INFO [Listener at localhost/46667] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-04-12 02:54:15,978 INFO [Listener at localhost/46667] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-04-12 02:54:15,978 INFO [Listener at localhost/46667] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-04-12 02:54:15,979 INFO [Listener at localhost/46667] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-04-12 02:54:15,979 DEBUG [Listener at localhost/46667] store.ProcedureTree(114): Procedure Procedure(pid=44, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[0, 1, 2] 2023-04-12 02:54:15,980 DEBUG [Listener at localhost/46667] store.ProcedureTree(114): Procedure Procedure(pid=45, ppid=44, class=org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure) stack ids=[3] 2023-04-12 02:54:15,980 DEBUG [Listener at localhost/46667] store.ProcedureTree(114): Procedure Procedure(pid=46, ppid=45, class=org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure) stack ids=[4, 5] 2023-04-12 02:54:15,980 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-04-12 02:54:15,981 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-04-12 02:54:15,981 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-04-12 02:54:15,981 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-04-12 02:54:15,982 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-12 02:54:15,982 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=cc9012ec0b656baa24d04368fbe026c4, daughterA=e2a9b8fe15a781798c0b49e48ff5aad7, daughterB=6f55b5d6a460bd8c7569dee22f049104 2023-04-12 02:54:15,982 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-12 02:54:15,982 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-12 02:54:15,983 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-04-12 02:54:15,983 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=28, state=SUCCESS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=315e13e868a7f68a3ce37802b2f6312d, daughterA=4e1092f3c00484f8f09c730861225d43, daughterB=291734c00d212fd203b04cf8872d4649 2023-04-12 02:54:15,983 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=35, state=SUCCESS; DisableTableProcedure table=testSplitWithoutPONR 2023-04-12 02:54:15,983 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=40, state=SUCCESS; DeleteTableProcedure table=testSplitWithoutPONR 2023-04-12 02:54:15,984 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=41, state=SUCCESS; CreateTableProcedure table=testRollbackAndDoubleExecution 2023-04-12 02:54:15,984 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(468): Loading pid=44, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=96cdae2f7638212730b694522b9a464a, daughterA=fdd18e9cf539c13d863267ce16031583, daughterB=578095173fbc35e181cce14464037c2e 2023-04-12 02:54:15,985 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(468): Loading pid=45, ppid=44, state=RUNNABLE:REGION_STATE_TRANSITION_CONFIRM_CLOSED; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=96cdae2f7638212730b694522b9a464a, UNASSIGN 2023-04-12 02:54:15,985 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(468): Loading pid=46, ppid=45, state=SUCCESS; CloseRegionProcedure 96cdae2f7638212730b694522b9a464a, server=jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:15,985 DEBUG [Listener at localhost/46667] procedure2.Procedure(966): pid=44, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=96cdae2f7638212730b694522b9a464a, daughterA=fdd18e9cf539c13d863267ce16031583, daughterB=578095173fbc35e181cce14464037c2e held the lock before restarting, call acquireLock to restore it. 2023-04-12 02:54:15,985 INFO [Listener at localhost/46667] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=96cdae2f7638212730b694522b9a464a, daughterA=fdd18e9cf539c13d863267ce16031583, daughterB=578095173fbc35e181cce14464037c2e 2023-04-12 02:54:15,985 INFO [Listener at localhost/46667] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=96cdae2f7638212730b694522b9a464a, daughterA=fdd18e9cf539c13d863267ce16031583, daughterB=578095173fbc35e181cce14464037c2e 2023-04-12 02:54:15,985 INFO [Listener at localhost/46667] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=96cdae2f7638212730b694522b9a464a, daughterA=fdd18e9cf539c13d863267ce16031583, daughterB=578095173fbc35e181cce14464037c2e 2023-04-12 02:54:15,985 DEBUG [Listener at localhost/46667] procedure2.Procedure(946): pid=45, ppid=44, state=RUNNABLE:REGION_STATE_TRANSITION_CONFIRM_CLOSED; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=96cdae2f7638212730b694522b9a464a, UNASSIGN didn't hold the lock before restarting, skip acquiring lock. 2023-04-12 02:54:15,985 DEBUG [Listener at localhost/46667] procedure2.Procedure(946): pid=46, ppid=45, state=SUCCESS; CloseRegionProcedure 96cdae2f7638212730b694522b9a464a, server=jenkins-hbase4.apache.org,44679,1681268031772 didn't hold the lock before restarting, skip acquiring lock. 2023-04-12 02:54:15,985 INFO [Listener at localhost/46667] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 6 msec 2023-04-12 02:54:15,988 DEBUG [Listener at localhost/46667] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-04-12 02:54:15,991 DEBUG [Listener at localhost/46667] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase4.apache.org,44679,1681268031772, table=hbase:meta, region=1588230740 2023-04-12 02:54:15,991 INFO [Listener at localhost/46667] assignment.AssignmentManager(305): Attach pid=45, ppid=44, state=RUNNABLE:REGION_STATE_TRANSITION_CONFIRM_CLOSED; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=96cdae2f7638212730b694522b9a464a, UNASSIGN to state=OFFLINE, location=null, table=testRollbackAndDoubleExecution, region=96cdae2f7638212730b694522b9a464a to restore RIT 2023-04-12 02:54:15,992 INFO [Listener at localhost/46667] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1681268085992 2023-04-12 02:54:15,993 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=45, ppid=44, state=RUNNABLE:REGION_STATE_TRANSITION_CONFIRM_CLOSED; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=96cdae2f7638212730b694522b9a464a, UNASSIGN 2023-04-12 02:54:15,993 DEBUG [Listener at localhost/46667] assignment.AssignmentManager(1517): Joining cluster... 2023-04-12 02:54:15,996 DEBUG [Listener at localhost/46667] assignment.RegionStateStore(147): Load hbase:meta entry region=1f255e980a31bb266f2206a30aa2ae3f, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,44679,1681268031772, regionLocation=jenkins-hbase4.apache.org,44679,1681268031772, openSeqNum=2 2023-04-12 02:54:15,996 DEBUG [Listener at localhost/46667] assignment.RegionStateStore(147): Load hbase:meta entry region=96cdae2f7638212730b694522b9a464a, regionState=CLOSED, lastHost=jenkins-hbase4.apache.org,44679,1681268031772, regionLocation=jenkins-hbase4.apache.org,44679,1681268031772, openSeqNum=2 2023-04-12 02:54:15,996 INFO [Listener at localhost/46667] assignment.AssignmentManager(1529): Number of RegionServers=3 2023-04-12 02:54:15,996 INFO [Listener at localhost/46667] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1681268115996 2023-04-12 02:54:15,996 INFO [Listener at localhost/46667] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1681268175996 2023-04-12 02:54:15,996 INFO [Listener at localhost/46667] assignment.AssignmentManager(1536): Joined the cluster in 3 msec 2023-04-12 02:54:15,997 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=45, ppid=44, state=RUNNABLE:REGION_STATE_TRANSITION_CONFIRM_CLOSED; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=96cdae2f7638212730b694522b9a464a, UNASSIGN 2023-04-12 02:54:15,998 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-04-12 02:54:16,002 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=45, resume processing ppid=44 2023-04-12 02:54:16,003 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=45, ppid=44, state=SUCCESS; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=96cdae2f7638212730b694522b9a464a, UNASSIGN in 829 msec 2023-04-12 02:54:16,006 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-04-12 02:54:16,006 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_CREATE_DAUGHTER_REGIONS, locked=true; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=96cdae2f7638212730b694522b9a464a, daughterA=fdd18e9cf539c13d863267ce16031583, daughterB=578095173fbc35e181cce14464037c2e 2023-04-12 02:54:16,006 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-04-12 02:54:16,006 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_CREATE_DAUGHTER_REGIONS, locked=true; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=96cdae2f7638212730b694522b9a464a, daughterA=fdd18e9cf539c13d863267ce16031583, daughterB=578095173fbc35e181cce14464037c2e at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-04-12 02:54:16,247 INFO [Listener at localhost/46667] procedure.MasterProcedureTestingUtility(417): Restart 4 exec state=pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_CREATE_DAUGHTER_REGIONS, locked=true; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=96cdae2f7638212730b694522b9a464a, daughterA=fdd18e9cf539c13d863267ce16031583, daughterB=578095173fbc35e181cce14464037c2e 2023-04-12 02:54:16,247 INFO [Listener at localhost/46667] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-04-12 02:54:16,247 INFO [Listener at localhost/46667] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-04-12 02:54:16,247 INFO [Listener at localhost/46667] assignment.AssignmentManager(315): Stopping assignment manager 2023-04-12 02:54:16,248 ERROR [Listener at localhost/46667] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-04-12 02:54:16,248 INFO [Listener at localhost/46667] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-04-12 02:54:16,248 INFO [Listener at localhost/46667] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-04-12 02:54:16,248 INFO [Listener at localhost/46667] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-04-12 02:54:16,248 INFO [Listener at localhost/46667] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-04-12 02:54:16,249 INFO [Listener at localhost/46667] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-04-12 02:54:16,250 DEBUG [Listener at localhost/46667] store.ProcedureTree(114): Procedure Procedure(pid=44, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[0, 1, 2] 2023-04-12 02:54:16,250 DEBUG [Listener at localhost/46667] store.ProcedureTree(114): Procedure Procedure(pid=45, ppid=44, class=org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure) stack ids=[3, 6] 2023-04-12 02:54:16,250 DEBUG [Listener at localhost/46667] store.ProcedureTree(114): Procedure Procedure(pid=46, ppid=45, class=org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure) stack ids=[4, 5] 2023-04-12 02:54:16,250 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-04-12 02:54:16,251 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-04-12 02:54:16,251 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-04-12 02:54:16,251 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-04-12 02:54:16,251 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-12 02:54:16,251 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=cc9012ec0b656baa24d04368fbe026c4, daughterA=e2a9b8fe15a781798c0b49e48ff5aad7, daughterB=6f55b5d6a460bd8c7569dee22f049104 2023-04-12 02:54:16,252 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-12 02:54:16,252 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-12 02:54:16,252 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-04-12 02:54:16,252 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=28, state=SUCCESS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=315e13e868a7f68a3ce37802b2f6312d, daughterA=4e1092f3c00484f8f09c730861225d43, daughterB=291734c00d212fd203b04cf8872d4649 2023-04-12 02:54:16,253 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=35, state=SUCCESS; DisableTableProcedure table=testSplitWithoutPONR 2023-04-12 02:54:16,253 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=40, state=SUCCESS; DeleteTableProcedure table=testSplitWithoutPONR 2023-04-12 02:54:16,253 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=41, state=SUCCESS; CreateTableProcedure table=testRollbackAndDoubleExecution 2023-04-12 02:54:16,254 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(468): Loading pid=44, state=RUNNABLE:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=96cdae2f7638212730b694522b9a464a, daughterA=fdd18e9cf539c13d863267ce16031583, daughterB=578095173fbc35e181cce14464037c2e 2023-04-12 02:54:16,254 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(468): Loading pid=45, ppid=44, state=SUCCESS; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=96cdae2f7638212730b694522b9a464a, UNASSIGN 2023-04-12 02:54:16,254 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(468): Loading pid=46, ppid=45, state=SUCCESS; CloseRegionProcedure 96cdae2f7638212730b694522b9a464a, server=jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:16,254 DEBUG [Listener at localhost/46667] procedure2.Procedure(946): pid=44, state=RUNNABLE:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=96cdae2f7638212730b694522b9a464a, daughterA=fdd18e9cf539c13d863267ce16031583, daughterB=578095173fbc35e181cce14464037c2e didn't hold the lock before restarting, skip acquiring lock. 2023-04-12 02:54:16,254 DEBUG [Listener at localhost/46667] procedure2.Procedure(946): pid=45, ppid=44, state=SUCCESS; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=96cdae2f7638212730b694522b9a464a, UNASSIGN didn't hold the lock before restarting, skip acquiring lock. 2023-04-12 02:54:16,254 DEBUG [Listener at localhost/46667] procedure2.Procedure(946): pid=46, ppid=45, state=SUCCESS; CloseRegionProcedure 96cdae2f7638212730b694522b9a464a, server=jenkins-hbase4.apache.org,44679,1681268031772 didn't hold the lock before restarting, skip acquiring lock. 2023-04-12 02:54:16,254 INFO [Listener at localhost/46667] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 5 msec 2023-04-12 02:54:16,255 DEBUG [Listener at localhost/46667] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-04-12 02:54:16,255 DEBUG [Listener at localhost/46667] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase4.apache.org,44679,1681268031772, table=hbase:meta, region=1588230740 2023-04-12 02:54:16,256 INFO [Listener at localhost/46667] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1681268086256 2023-04-12 02:54:16,257 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=44, state=RUNNABLE:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=96cdae2f7638212730b694522b9a464a, daughterA=fdd18e9cf539c13d863267ce16031583, daughterB=578095173fbc35e181cce14464037c2e 2023-04-12 02:54:16,257 DEBUG [Listener at localhost/46667] assignment.AssignmentManager(1517): Joining cluster... 2023-04-12 02:54:16,260 DEBUG [Listener at localhost/46667] assignment.RegionStateStore(147): Load hbase:meta entry region=1f255e980a31bb266f2206a30aa2ae3f, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,44679,1681268031772, regionLocation=jenkins-hbase4.apache.org,44679,1681268031772, openSeqNum=2 2023-04-12 02:54:16,260 DEBUG [Listener at localhost/46667] assignment.RegionStateStore(147): Load hbase:meta entry region=96cdae2f7638212730b694522b9a464a, regionState=CLOSED, lastHost=jenkins-hbase4.apache.org,44679,1681268031772, regionLocation=jenkins-hbase4.apache.org,44679,1681268031772, openSeqNum=2 2023-04-12 02:54:16,260 INFO [Listener at localhost/46667] assignment.AssignmentManager(1529): Number of RegionServers=3 2023-04-12 02:54:16,260 INFO [Listener at localhost/46667] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1681268116260 2023-04-12 02:54:16,260 INFO [Listener at localhost/46667] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1681268176260 2023-04-12 02:54:16,261 INFO [Listener at localhost/46667] assignment.AssignmentManager(1536): Joined the cluster in 3 msec 2023-04-12 02:54:16,261 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=RUNNABLE:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=96cdae2f7638212730b694522b9a464a, daughterA=fdd18e9cf539c13d863267ce16031583, daughterB=578095173fbc35e181cce14464037c2e 2023-04-12 02:54:16,261 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=RUNNABLE:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=96cdae2f7638212730b694522b9a464a, daughterA=fdd18e9cf539c13d863267ce16031583, daughterB=578095173fbc35e181cce14464037c2e 2023-04-12 02:54:16,261 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=RUNNABLE:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=96cdae2f7638212730b694522b9a464a, daughterA=fdd18e9cf539c13d863267ce16031583, daughterB=578095173fbc35e181cce14464037c2e 2023-04-12 02:54:16,266 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-04-12 02:54:16,276 INFO [PEWorker-1] assignment.SplitTableRegionProcedure(694): pid=44 splitting 14 storefiles, region=96cdae2f7638212730b694522b9a464a, threads=14 2023-04-12 02:54:16,291 DEBUG [StoreFileSplitter-pool-0] assignment.SplitTableRegionProcedure(776): pid=44 splitting started for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/cf2/11c4addd69d049beb33f17d53e40ab23 for region: 96cdae2f7638212730b694522b9a464a 2023-04-12 02:54:16,292 DEBUG [StoreFileSplitter-pool-1] assignment.SplitTableRegionProcedure(776): pid=44 splitting started for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/cf2/28ceff5b76a942bd8ce4bf8f9ffcc8c9 for region: 96cdae2f7638212730b694522b9a464a 2023-04-12 02:54:16,304 DEBUG [StoreFileSplitter-pool-2] assignment.SplitTableRegionProcedure(776): pid=44 splitting started for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/cf2/68757554e4ad4422867a2933985cb078 for region: 96cdae2f7638212730b694522b9a464a 2023-04-12 02:54:16,308 DEBUG [StoreFileSplitter-pool-3] assignment.SplitTableRegionProcedure(776): pid=44 splitting started for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/cf2/6c612c6ffd3d42eab35c5ae09327ecf8 for region: 96cdae2f7638212730b694522b9a464a 2023-04-12 02:54:16,309 DEBUG [StoreFileSplitter-pool-4] assignment.SplitTableRegionProcedure(776): pid=44 splitting started for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/cf2/a0a6721d34874582969e0d913c5a3410 for region: 96cdae2f7638212730b694522b9a464a 2023-04-12 02:54:16,316 DEBUG [StoreFileSplitter-pool-5] assignment.SplitTableRegionProcedure(776): pid=44 splitting started for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/cf2/cd4efe4b5ad64256947de9915b5f6552 for region: 96cdae2f7638212730b694522b9a464a 2023-04-12 02:54:16,324 DEBUG [StoreFileSplitter-pool-6] assignment.SplitTableRegionProcedure(776): pid=44 splitting started for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/cf2/f66940ce4359474eac6c223cf1a8b486 for region: 96cdae2f7638212730b694522b9a464a 2023-04-12 02:54:16,324 DEBUG [StoreFileSplitter-pool-7] assignment.SplitTableRegionProcedure(776): pid=44 splitting started for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/cf1/0693b446d83544e4a9700be8c15de027 for region: 96cdae2f7638212730b694522b9a464a 2023-04-12 02:54:16,343 DEBUG [StoreFileSplitter-pool-8] assignment.SplitTableRegionProcedure(776): pid=44 splitting started for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/cf1/0864a86174d140c0a957973bc4f9f7dc for region: 96cdae2f7638212730b694522b9a464a 2023-04-12 02:54:16,348 DEBUG [StoreFileSplitter-pool-9] assignment.SplitTableRegionProcedure(776): pid=44 splitting started for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/cf1/091b8b6c71524b5ea1657e72778f8478 for region: 96cdae2f7638212730b694522b9a464a 2023-04-12 02:54:16,351 DEBUG [StoreFileSplitter-pool-10] assignment.SplitTableRegionProcedure(776): pid=44 splitting started for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/cf1/177232c91ccf4f5d91bf18c2d4d5022f for region: 96cdae2f7638212730b694522b9a464a 2023-04-12 02:54:16,359 DEBUG [StoreFileSplitter-pool-11] assignment.SplitTableRegionProcedure(776): pid=44 splitting started for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/cf1/6fbe235c9d994e648689a95a2833784a for region: 96cdae2f7638212730b694522b9a464a 2023-04-12 02:54:16,362 DEBUG [StoreFileSplitter-pool-12] assignment.SplitTableRegionProcedure(776): pid=44 splitting started for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/cf1/a3b5b9d8fa954923ae97ea97984ade02 for region: 96cdae2f7638212730b694522b9a464a 2023-04-12 02:54:16,362 DEBUG [StoreFileSplitter-pool-13] assignment.SplitTableRegionProcedure(776): pid=44 splitting started for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/cf1/b5b939a0ed784022a8f999fa3edea7b9 for region: 96cdae2f7638212730b694522b9a464a 2023-04-12 02:54:16,720 DEBUG [StoreFileSplitter-pool-2] assignment.SplitTableRegionProcedure(787): pid=44 splitting complete for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/cf2/68757554e4ad4422867a2933985cb078 for region: 96cdae2f7638212730b694522b9a464a 2023-04-12 02:54:16,724 DEBUG [StoreFileSplitter-pool-5] assignment.SplitTableRegionProcedure(787): pid=44 splitting complete for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/cf2/cd4efe4b5ad64256947de9915b5f6552 for region: 96cdae2f7638212730b694522b9a464a 2023-04-12 02:54:16,728 DEBUG [StoreFileSplitter-pool-0] assignment.SplitTableRegionProcedure(787): pid=44 splitting complete for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/cf2/11c4addd69d049beb33f17d53e40ab23 for region: 96cdae2f7638212730b694522b9a464a 2023-04-12 02:54:16,729 DEBUG [StoreFileSplitter-pool-12] assignment.SplitTableRegionProcedure(787): pid=44 splitting complete for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/cf1/a3b5b9d8fa954923ae97ea97984ade02 for region: 96cdae2f7638212730b694522b9a464a 2023-04-12 02:54:16,730 DEBUG [StoreFileSplitter-pool-7] assignment.SplitTableRegionProcedure(787): pid=44 splitting complete for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/cf1/0693b446d83544e4a9700be8c15de027 for region: 96cdae2f7638212730b694522b9a464a 2023-04-12 02:54:16,730 DEBUG [StoreFileSplitter-pool-4] assignment.SplitTableRegionProcedure(787): pid=44 splitting complete for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/cf2/a0a6721d34874582969e0d913c5a3410 for region: 96cdae2f7638212730b694522b9a464a 2023-04-12 02:54:16,741 DEBUG [StoreFileSplitter-pool-11] assignment.SplitTableRegionProcedure(787): pid=44 splitting complete for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/cf1/6fbe235c9d994e648689a95a2833784a for region: 96cdae2f7638212730b694522b9a464a 2023-04-12 02:54:16,741 DEBUG [StoreFileSplitter-pool-9] assignment.SplitTableRegionProcedure(787): pid=44 splitting complete for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/cf1/091b8b6c71524b5ea1657e72778f8478 for region: 96cdae2f7638212730b694522b9a464a 2023-04-12 02:54:16,751 DEBUG [StoreFileSplitter-pool-3] assignment.SplitTableRegionProcedure(787): pid=44 splitting complete for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/cf2/6c612c6ffd3d42eab35c5ae09327ecf8 for region: 96cdae2f7638212730b694522b9a464a 2023-04-12 02:54:16,753 DEBUG [StoreFileSplitter-pool-1] assignment.SplitTableRegionProcedure(787): pid=44 splitting complete for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/cf2/28ceff5b76a942bd8ce4bf8f9ffcc8c9 for region: 96cdae2f7638212730b694522b9a464a 2023-04-12 02:54:16,756 DEBUG [StoreFileSplitter-pool-6] assignment.SplitTableRegionProcedure(787): pid=44 splitting complete for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/cf2/f66940ce4359474eac6c223cf1a8b486 for region: 96cdae2f7638212730b694522b9a464a 2023-04-12 02:54:16,756 DEBUG [StoreFileSplitter-pool-13] assignment.SplitTableRegionProcedure(787): pid=44 splitting complete for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/cf1/b5b939a0ed784022a8f999fa3edea7b9 for region: 96cdae2f7638212730b694522b9a464a 2023-04-12 02:54:16,759 DEBUG [StoreFileSplitter-pool-8] assignment.SplitTableRegionProcedure(787): pid=44 splitting complete for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/cf1/0864a86174d140c0a957973bc4f9f7dc for region: 96cdae2f7638212730b694522b9a464a 2023-04-12 02:54:16,759 DEBUG [StoreFileSplitter-pool-10] assignment.SplitTableRegionProcedure(787): pid=44 splitting complete for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/cf1/177232c91ccf4f5d91bf18c2d4d5022f for region: 96cdae2f7638212730b694522b9a464a 2023-04-12 02:54:16,759 DEBUG [PEWorker-1] assignment.SplitTableRegionProcedure(755): pid=44 split storefiles for region 96cdae2f7638212730b694522b9a464a Daughter A: 14 storefiles, Daughter B: 14 storefiles. 2023-04-12 02:54:16,801 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-04-12 02:54:16,802 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_WRITE_MAX_SEQUENCE_ID_FILE, locked=true; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=96cdae2f7638212730b694522b9a464a, daughterA=fdd18e9cf539c13d863267ce16031583, daughterB=578095173fbc35e181cce14464037c2e 2023-04-12 02:54:16,802 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-04-12 02:54:16,802 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_WRITE_MAX_SEQUENCE_ID_FILE, locked=true; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=96cdae2f7638212730b694522b9a464a, daughterA=fdd18e9cf539c13d863267ce16031583, daughterB=578095173fbc35e181cce14464037c2e at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-04-12 02:54:17,011 INFO [Listener at localhost/46667] procedure.MasterProcedureTestingUtility(417): Restart 5 exec state=pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_WRITE_MAX_SEQUENCE_ID_FILE, locked=true; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=96cdae2f7638212730b694522b9a464a, daughterA=fdd18e9cf539c13d863267ce16031583, daughterB=578095173fbc35e181cce14464037c2e 2023-04-12 02:54:17,011 INFO [Listener at localhost/46667] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-04-12 02:54:17,011 INFO [Listener at localhost/46667] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-04-12 02:54:17,011 INFO [Listener at localhost/46667] assignment.AssignmentManager(315): Stopping assignment manager 2023-04-12 02:54:17,011 ERROR [Listener at localhost/46667] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-04-12 02:54:17,012 INFO [Listener at localhost/46667] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-04-12 02:54:17,012 INFO [Listener at localhost/46667] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-04-12 02:54:17,012 INFO [Listener at localhost/46667] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-04-12 02:54:17,012 INFO [Listener at localhost/46667] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-04-12 02:54:17,013 INFO [Listener at localhost/46667] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-04-12 02:54:17,013 DEBUG [Listener at localhost/46667] store.ProcedureTree(114): Procedure Procedure(pid=44, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[0, 1, 2, 7] 2023-04-12 02:54:17,014 DEBUG [Listener at localhost/46667] store.ProcedureTree(114): Procedure Procedure(pid=45, ppid=44, class=org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure) stack ids=[3, 6] 2023-04-12 02:54:17,014 DEBUG [Listener at localhost/46667] store.ProcedureTree(114): Procedure Procedure(pid=46, ppid=45, class=org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure) stack ids=[4, 5] 2023-04-12 02:54:17,014 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-04-12 02:54:17,015 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-04-12 02:54:17,015 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-04-12 02:54:17,015 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-04-12 02:54:17,015 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-12 02:54:17,016 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=cc9012ec0b656baa24d04368fbe026c4, daughterA=e2a9b8fe15a781798c0b49e48ff5aad7, daughterB=6f55b5d6a460bd8c7569dee22f049104 2023-04-12 02:54:17,016 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-12 02:54:17,016 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-12 02:54:17,016 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-04-12 02:54:17,017 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=28, state=SUCCESS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=315e13e868a7f68a3ce37802b2f6312d, daughterA=4e1092f3c00484f8f09c730861225d43, daughterB=291734c00d212fd203b04cf8872d4649 2023-04-12 02:54:17,017 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=35, state=SUCCESS; DisableTableProcedure table=testSplitWithoutPONR 2023-04-12 02:54:17,017 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=40, state=SUCCESS; DeleteTableProcedure table=testSplitWithoutPONR 2023-04-12 02:54:17,017 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=41, state=SUCCESS; CreateTableProcedure table=testRollbackAndDoubleExecution 2023-04-12 02:54:17,018 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(468): Loading pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_CREATE_DAUGHTER_REGIONS; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=96cdae2f7638212730b694522b9a464a, daughterA=fdd18e9cf539c13d863267ce16031583, daughterB=578095173fbc35e181cce14464037c2e 2023-04-12 02:54:17,018 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(468): Loading pid=45, ppid=44, state=SUCCESS; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=96cdae2f7638212730b694522b9a464a, UNASSIGN 2023-04-12 02:54:17,019 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(468): Loading pid=46, ppid=45, state=SUCCESS; CloseRegionProcedure 96cdae2f7638212730b694522b9a464a, server=jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:17,019 DEBUG [Listener at localhost/46667] procedure2.Procedure(966): pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_CREATE_DAUGHTER_REGIONS; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=96cdae2f7638212730b694522b9a464a, daughterA=fdd18e9cf539c13d863267ce16031583, daughterB=578095173fbc35e181cce14464037c2e held the lock before restarting, call acquireLock to restore it. 2023-04-12 02:54:17,019 INFO [Listener at localhost/46667] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_CREATE_DAUGHTER_REGIONS; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=96cdae2f7638212730b694522b9a464a, daughterA=fdd18e9cf539c13d863267ce16031583, daughterB=578095173fbc35e181cce14464037c2e 2023-04-12 02:54:17,019 INFO [Listener at localhost/46667] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_CREATE_DAUGHTER_REGIONS; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=96cdae2f7638212730b694522b9a464a, daughterA=fdd18e9cf539c13d863267ce16031583, daughterB=578095173fbc35e181cce14464037c2e 2023-04-12 02:54:17,019 INFO [Listener at localhost/46667] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_CREATE_DAUGHTER_REGIONS; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=96cdae2f7638212730b694522b9a464a, daughterA=fdd18e9cf539c13d863267ce16031583, daughterB=578095173fbc35e181cce14464037c2e 2023-04-12 02:54:17,019 DEBUG [Listener at localhost/46667] procedure2.Procedure(946): pid=45, ppid=44, state=SUCCESS; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=96cdae2f7638212730b694522b9a464a, UNASSIGN didn't hold the lock before restarting, skip acquiring lock. 2023-04-12 02:54:17,019 DEBUG [Listener at localhost/46667] procedure2.Procedure(946): pid=46, ppid=45, state=SUCCESS; CloseRegionProcedure 96cdae2f7638212730b694522b9a464a, server=jenkins-hbase4.apache.org,44679,1681268031772 didn't hold the lock before restarting, skip acquiring lock. 2023-04-12 02:54:17,019 INFO [Listener at localhost/46667] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 6 msec 2023-04-12 02:54:17,020 DEBUG [Listener at localhost/46667] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-04-12 02:54:17,021 DEBUG [Listener at localhost/46667] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase4.apache.org,44679,1681268031772, table=hbase:meta, region=1588230740 2023-04-12 02:54:17,021 INFO [Listener at localhost/46667] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1681268087021 2023-04-12 02:54:17,022 DEBUG [Listener at localhost/46667] assignment.AssignmentManager(1517): Joining cluster... 2023-04-12 02:54:17,022 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_CREATE_DAUGHTER_REGIONS; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=96cdae2f7638212730b694522b9a464a, daughterA=fdd18e9cf539c13d863267ce16031583, daughterB=578095173fbc35e181cce14464037c2e 2023-04-12 02:54:17,025 DEBUG [Listener at localhost/46667] assignment.RegionStateStore(147): Load hbase:meta entry region=1f255e980a31bb266f2206a30aa2ae3f, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,44679,1681268031772, regionLocation=jenkins-hbase4.apache.org,44679,1681268031772, openSeqNum=2 2023-04-12 02:54:17,025 DEBUG [Listener at localhost/46667] assignment.RegionStateStore(147): Load hbase:meta entry region=96cdae2f7638212730b694522b9a464a, regionState=CLOSED, lastHost=jenkins-hbase4.apache.org,44679,1681268031772, regionLocation=jenkins-hbase4.apache.org,44679,1681268031772, openSeqNum=2 2023-04-12 02:54:17,025 INFO [Listener at localhost/46667] assignment.AssignmentManager(1529): Number of RegionServers=3 2023-04-12 02:54:17,025 INFO [Listener at localhost/46667] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1681268117025 2023-04-12 02:54:17,025 INFO [Listener at localhost/46667] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1681268177025 2023-04-12 02:54:17,025 INFO [Listener at localhost/46667] assignment.AssignmentManager(1536): Joined the cluster in 2 msec 2023-04-12 02:54:17,034 INFO [PEWorker-1] assignment.SplitTableRegionProcedure(694): pid=44 splitting 14 storefiles, region=96cdae2f7638212730b694522b9a464a, threads=14 2023-04-12 02:54:17,035 DEBUG [StoreFileSplitter-pool-0] assignment.SplitTableRegionProcedure(776): pid=44 splitting started for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/cf2/11c4addd69d049beb33f17d53e40ab23 for region: 96cdae2f7638212730b694522b9a464a 2023-04-12 02:54:17,035 DEBUG [StoreFileSplitter-pool-1] assignment.SplitTableRegionProcedure(776): pid=44 splitting started for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/cf2/28ceff5b76a942bd8ce4bf8f9ffcc8c9 for region: 96cdae2f7638212730b694522b9a464a 2023-04-12 02:54:17,035 DEBUG [StoreFileSplitter-pool-2] assignment.SplitTableRegionProcedure(776): pid=44 splitting started for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/cf2/68757554e4ad4422867a2933985cb078 for region: 96cdae2f7638212730b694522b9a464a 2023-04-12 02:54:17,035 DEBUG [StoreFileSplitter-pool-3] assignment.SplitTableRegionProcedure(776): pid=44 splitting started for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/cf2/6c612c6ffd3d42eab35c5ae09327ecf8 for region: 96cdae2f7638212730b694522b9a464a 2023-04-12 02:54:17,035 DEBUG [StoreFileSplitter-pool-4] assignment.SplitTableRegionProcedure(776): pid=44 splitting started for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/cf2/a0a6721d34874582969e0d913c5a3410 for region: 96cdae2f7638212730b694522b9a464a 2023-04-12 02:54:17,035 DEBUG [StoreFileSplitter-pool-5] assignment.SplitTableRegionProcedure(776): pid=44 splitting started for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/cf2/cd4efe4b5ad64256947de9915b5f6552 for region: 96cdae2f7638212730b694522b9a464a 2023-04-12 02:54:17,035 DEBUG [StoreFileSplitter-pool-6] assignment.SplitTableRegionProcedure(776): pid=44 splitting started for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/cf2/f66940ce4359474eac6c223cf1a8b486 for region: 96cdae2f7638212730b694522b9a464a 2023-04-12 02:54:17,035 DEBUG [StoreFileSplitter-pool-7] assignment.SplitTableRegionProcedure(776): pid=44 splitting started for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/cf1/0693b446d83544e4a9700be8c15de027 for region: 96cdae2f7638212730b694522b9a464a 2023-04-12 02:54:17,036 DEBUG [StoreFileSplitter-pool-8] assignment.SplitTableRegionProcedure(776): pid=44 splitting started for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/cf1/0864a86174d140c0a957973bc4f9f7dc for region: 96cdae2f7638212730b694522b9a464a 2023-04-12 02:54:17,036 DEBUG [StoreFileSplitter-pool-9] assignment.SplitTableRegionProcedure(776): pid=44 splitting started for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/cf1/091b8b6c71524b5ea1657e72778f8478 for region: 96cdae2f7638212730b694522b9a464a 2023-04-12 02:54:17,036 DEBUG [StoreFileSplitter-pool-10] assignment.SplitTableRegionProcedure(776): pid=44 splitting started for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/cf1/177232c91ccf4f5d91bf18c2d4d5022f for region: 96cdae2f7638212730b694522b9a464a 2023-04-12 02:54:17,039 DEBUG [StoreFileSplitter-pool-11] assignment.SplitTableRegionProcedure(776): pid=44 splitting started for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/cf1/6fbe235c9d994e648689a95a2833784a for region: 96cdae2f7638212730b694522b9a464a 2023-04-12 02:54:17,039 DEBUG [StoreFileSplitter-pool-12] assignment.SplitTableRegionProcedure(776): pid=44 splitting started for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/cf1/a3b5b9d8fa954923ae97ea97984ade02 for region: 96cdae2f7638212730b694522b9a464a 2023-04-12 02:54:17,039 DEBUG [StoreFileSplitter-pool-13] assignment.SplitTableRegionProcedure(776): pid=44 splitting started for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/cf1/b5b939a0ed784022a8f999fa3edea7b9 for region: 96cdae2f7638212730b694522b9a464a 2023-04-12 02:54:17,147 DEBUG [StoreFileSplitter-pool-3] assignment.SplitTableRegionProcedure(787): pid=44 splitting complete for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/cf2/6c612c6ffd3d42eab35c5ae09327ecf8 for region: 96cdae2f7638212730b694522b9a464a 2023-04-12 02:54:17,160 DEBUG [StoreFileSplitter-pool-9] assignment.SplitTableRegionProcedure(787): pid=44 splitting complete for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/cf1/091b8b6c71524b5ea1657e72778f8478 for region: 96cdae2f7638212730b694522b9a464a 2023-04-12 02:54:17,160 DEBUG [StoreFileSplitter-pool-10] assignment.SplitTableRegionProcedure(787): pid=44 splitting complete for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/cf1/177232c91ccf4f5d91bf18c2d4d5022f for region: 96cdae2f7638212730b694522b9a464a 2023-04-12 02:54:17,184 DEBUG [StoreFileSplitter-pool-7] assignment.SplitTableRegionProcedure(787): pid=44 splitting complete for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/cf1/0693b446d83544e4a9700be8c15de027 for region: 96cdae2f7638212730b694522b9a464a 2023-04-12 02:54:17,186 DEBUG [StoreFileSplitter-pool-1] assignment.SplitTableRegionProcedure(787): pid=44 splitting complete for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/cf2/28ceff5b76a942bd8ce4bf8f9ffcc8c9 for region: 96cdae2f7638212730b694522b9a464a 2023-04-12 02:54:17,196 DEBUG [StoreFileSplitter-pool-0] assignment.SplitTableRegionProcedure(787): pid=44 splitting complete for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/cf2/11c4addd69d049beb33f17d53e40ab23 for region: 96cdae2f7638212730b694522b9a464a 2023-04-12 02:54:17,201 DEBUG [StoreFileSplitter-pool-8] assignment.SplitTableRegionProcedure(787): pid=44 splitting complete for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/cf1/0864a86174d140c0a957973bc4f9f7dc for region: 96cdae2f7638212730b694522b9a464a 2023-04-12 02:54:17,204 DEBUG [StoreFileSplitter-pool-6] assignment.SplitTableRegionProcedure(787): pid=44 splitting complete for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/cf2/f66940ce4359474eac6c223cf1a8b486 for region: 96cdae2f7638212730b694522b9a464a 2023-04-12 02:54:17,205 DEBUG [StoreFileSplitter-pool-12] assignment.SplitTableRegionProcedure(787): pid=44 splitting complete for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/cf1/a3b5b9d8fa954923ae97ea97984ade02 for region: 96cdae2f7638212730b694522b9a464a 2023-04-12 02:54:17,206 DEBUG [StoreFileSplitter-pool-13] assignment.SplitTableRegionProcedure(787): pid=44 splitting complete for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/cf1/b5b939a0ed784022a8f999fa3edea7b9 for region: 96cdae2f7638212730b694522b9a464a 2023-04-12 02:54:17,207 DEBUG [StoreFileSplitter-pool-2] assignment.SplitTableRegionProcedure(787): pid=44 splitting complete for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/cf2/68757554e4ad4422867a2933985cb078 for region: 96cdae2f7638212730b694522b9a464a 2023-04-12 02:54:17,511 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-12 02:54:17,530 DEBUG [StoreFileSplitter-pool-4] assignment.SplitTableRegionProcedure(787): pid=44 splitting complete for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/cf2/a0a6721d34874582969e0d913c5a3410 for region: 96cdae2f7638212730b694522b9a464a 2023-04-12 02:54:17,533 DEBUG [StoreFileSplitter-pool-11] assignment.SplitTableRegionProcedure(787): pid=44 splitting complete for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/cf1/6fbe235c9d994e648689a95a2833784a for region: 96cdae2f7638212730b694522b9a464a 2023-04-12 02:54:17,551 DEBUG [StoreFileSplitter-pool-5] assignment.SplitTableRegionProcedure(787): pid=44 splitting complete for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/cf2/cd4efe4b5ad64256947de9915b5f6552 for region: 96cdae2f7638212730b694522b9a464a 2023-04-12 02:54:17,551 DEBUG [PEWorker-1] assignment.SplitTableRegionProcedure(755): pid=44 split storefiles for region 96cdae2f7638212730b694522b9a464a Daughter A: 14 storefiles, Daughter B: 14 storefiles. 2023-04-12 02:54:17,590 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-04-12 02:54:17,595 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/fdd18e9cf539c13d863267ce16031583/recovered.edits/85.seqid, newMaxSeqId=85, maxSeqId=-1 2023-04-12 02:54:17,596 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/578095173fbc35e181cce14464037c2e/recovered.edits/85.seqid, newMaxSeqId=85, maxSeqId=-1 2023-04-12 02:54:17,597 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-04-12 02:54:17,597 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_BEFORE_META, locked=true; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=96cdae2f7638212730b694522b9a464a, daughterA=fdd18e9cf539c13d863267ce16031583, daughterB=578095173fbc35e181cce14464037c2e 2023-04-12 02:54:17,597 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-04-12 02:54:17,597 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_BEFORE_META, locked=true; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=96cdae2f7638212730b694522b9a464a, daughterA=fdd18e9cf539c13d863267ce16031583, daughterB=578095173fbc35e181cce14464037c2e at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-04-12 02:54:17,775 INFO [Listener at localhost/46667] procedure.MasterProcedureTestingUtility(417): Restart 6 exec state=pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_BEFORE_META, locked=true; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=96cdae2f7638212730b694522b9a464a, daughterA=fdd18e9cf539c13d863267ce16031583, daughterB=578095173fbc35e181cce14464037c2e 2023-04-12 02:54:17,776 INFO [Listener at localhost/46667] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-04-12 02:54:17,776 INFO [Listener at localhost/46667] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-04-12 02:54:17,776 INFO [Listener at localhost/46667] assignment.AssignmentManager(315): Stopping assignment manager 2023-04-12 02:54:17,777 ERROR [Listener at localhost/46667] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-04-12 02:54:17,777 INFO [Listener at localhost/46667] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-04-12 02:54:17,777 INFO [Listener at localhost/46667] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-04-12 02:54:17,778 INFO [Listener at localhost/46667] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-04-12 02:54:17,778 INFO [Listener at localhost/46667] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-04-12 02:54:17,779 INFO [Listener at localhost/46667] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-04-12 02:54:17,780 DEBUG [Listener at localhost/46667] store.ProcedureTree(114): Procedure Procedure(pid=44, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[0, 1, 2, 7, 8] 2023-04-12 02:54:17,780 DEBUG [Listener at localhost/46667] store.ProcedureTree(114): Procedure Procedure(pid=45, ppid=44, class=org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure) stack ids=[3, 6] 2023-04-12 02:54:17,780 DEBUG [Listener at localhost/46667] store.ProcedureTree(114): Procedure Procedure(pid=46, ppid=45, class=org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure) stack ids=[4, 5] 2023-04-12 02:54:17,780 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-04-12 02:54:17,781 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-04-12 02:54:17,781 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-04-12 02:54:17,781 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-04-12 02:54:17,782 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-12 02:54:17,782 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=cc9012ec0b656baa24d04368fbe026c4, daughterA=e2a9b8fe15a781798c0b49e48ff5aad7, daughterB=6f55b5d6a460bd8c7569dee22f049104 2023-04-12 02:54:17,782 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-12 02:54:17,783 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-12 02:54:17,783 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-04-12 02:54:17,783 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=28, state=SUCCESS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=315e13e868a7f68a3ce37802b2f6312d, daughterA=4e1092f3c00484f8f09c730861225d43, daughterB=291734c00d212fd203b04cf8872d4649 2023-04-12 02:54:17,784 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=35, state=SUCCESS; DisableTableProcedure table=testSplitWithoutPONR 2023-04-12 02:54:17,784 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=40, state=SUCCESS; DeleteTableProcedure table=testSplitWithoutPONR 2023-04-12 02:54:17,784 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=41, state=SUCCESS; CreateTableProcedure table=testRollbackAndDoubleExecution 2023-04-12 02:54:17,785 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(468): Loading pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_WRITE_MAX_SEQUENCE_ID_FILE; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=96cdae2f7638212730b694522b9a464a, daughterA=fdd18e9cf539c13d863267ce16031583, daughterB=578095173fbc35e181cce14464037c2e 2023-04-12 02:54:17,785 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(468): Loading pid=45, ppid=44, state=SUCCESS; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=96cdae2f7638212730b694522b9a464a, UNASSIGN 2023-04-12 02:54:17,785 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(468): Loading pid=46, ppid=45, state=SUCCESS; CloseRegionProcedure 96cdae2f7638212730b694522b9a464a, server=jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:17,788 DEBUG [Listener at localhost/46667] procedure2.Procedure(966): pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_WRITE_MAX_SEQUENCE_ID_FILE; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=96cdae2f7638212730b694522b9a464a, daughterA=fdd18e9cf539c13d863267ce16031583, daughterB=578095173fbc35e181cce14464037c2e held the lock before restarting, call acquireLock to restore it. 2023-04-12 02:54:17,788 INFO [Listener at localhost/46667] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_WRITE_MAX_SEQUENCE_ID_FILE; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=96cdae2f7638212730b694522b9a464a, daughterA=fdd18e9cf539c13d863267ce16031583, daughterB=578095173fbc35e181cce14464037c2e 2023-04-12 02:54:17,788 INFO [Listener at localhost/46667] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_WRITE_MAX_SEQUENCE_ID_FILE; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=96cdae2f7638212730b694522b9a464a, daughterA=fdd18e9cf539c13d863267ce16031583, daughterB=578095173fbc35e181cce14464037c2e 2023-04-12 02:54:17,788 INFO [Listener at localhost/46667] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_WRITE_MAX_SEQUENCE_ID_FILE; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=96cdae2f7638212730b694522b9a464a, daughterA=fdd18e9cf539c13d863267ce16031583, daughterB=578095173fbc35e181cce14464037c2e 2023-04-12 02:54:17,789 DEBUG [Listener at localhost/46667] procedure2.Procedure(946): pid=45, ppid=44, state=SUCCESS; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=96cdae2f7638212730b694522b9a464a, UNASSIGN didn't hold the lock before restarting, skip acquiring lock. 2023-04-12 02:54:17,789 DEBUG [Listener at localhost/46667] procedure2.Procedure(946): pid=46, ppid=45, state=SUCCESS; CloseRegionProcedure 96cdae2f7638212730b694522b9a464a, server=jenkins-hbase4.apache.org,44679,1681268031772 didn't hold the lock before restarting, skip acquiring lock. 2023-04-12 02:54:17,789 INFO [Listener at localhost/46667] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 10 msec 2023-04-12 02:54:17,796 DEBUG [Listener at localhost/46667] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-04-12 02:54:17,797 DEBUG [Listener at localhost/46667] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase4.apache.org,44679,1681268031772, table=hbase:meta, region=1588230740 2023-04-12 02:54:17,800 INFO [Listener at localhost/46667] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1681268087800 2023-04-12 02:54:17,802 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_WRITE_MAX_SEQUENCE_ID_FILE; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=96cdae2f7638212730b694522b9a464a, daughterA=fdd18e9cf539c13d863267ce16031583, daughterB=578095173fbc35e181cce14464037c2e 2023-04-12 02:54:17,802 DEBUG [Listener at localhost/46667] assignment.AssignmentManager(1517): Joining cluster... 2023-04-12 02:54:17,804 DEBUG [Listener at localhost/46667] assignment.RegionStateStore(147): Load hbase:meta entry region=1f255e980a31bb266f2206a30aa2ae3f, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,44679,1681268031772, regionLocation=jenkins-hbase4.apache.org,44679,1681268031772, openSeqNum=2 2023-04-12 02:54:17,805 DEBUG [Listener at localhost/46667] assignment.RegionStateStore(147): Load hbase:meta entry region=96cdae2f7638212730b694522b9a464a, regionState=CLOSED, lastHost=jenkins-hbase4.apache.org,44679,1681268031772, regionLocation=jenkins-hbase4.apache.org,44679,1681268031772, openSeqNum=2 2023-04-12 02:54:17,805 INFO [Listener at localhost/46667] assignment.AssignmentManager(1529): Number of RegionServers=3 2023-04-12 02:54:17,805 INFO [Listener at localhost/46667] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1681268117805 2023-04-12 02:54:17,805 INFO [Listener at localhost/46667] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1681268177805 2023-04-12 02:54:17,805 INFO [Listener at localhost/46667] assignment.AssignmentManager(1536): Joined the cluster in 3 msec 2023-04-12 02:54:17,808 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-04-12 02:54:17,812 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-04-12 02:54:17,812 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_UPDATE_META, locked=true; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=96cdae2f7638212730b694522b9a464a, daughterA=fdd18e9cf539c13d863267ce16031583, daughterB=578095173fbc35e181cce14464037c2e 2023-04-12 02:54:17,812 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-04-12 02:54:17,812 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_UPDATE_META, locked=true; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=96cdae2f7638212730b694522b9a464a, daughterA=fdd18e9cf539c13d863267ce16031583, daughterB=578095173fbc35e181cce14464037c2e at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-04-12 02:54:18,056 INFO [Listener at localhost/46667] procedure.MasterProcedureTestingUtility(509): Restart 0 rollback state: pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_UPDATE_META, locked=true; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=96cdae2f7638212730b694522b9a464a, daughterA=fdd18e9cf539c13d863267ce16031583, daughterB=578095173fbc35e181cce14464037c2e 2023-04-12 02:54:18,056 INFO [Listener at localhost/46667] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-04-12 02:54:18,056 INFO [Listener at localhost/46667] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-04-12 02:54:18,056 INFO [Listener at localhost/46667] assignment.AssignmentManager(315): Stopping assignment manager 2023-04-12 02:54:18,057 ERROR [Listener at localhost/46667] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-04-12 02:54:18,057 INFO [Listener at localhost/46667] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-04-12 02:54:18,057 INFO [Listener at localhost/46667] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-04-12 02:54:18,057 INFO [Listener at localhost/46667] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-04-12 02:54:18,057 INFO [Listener at localhost/46667] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-04-12 02:54:18,058 INFO [Listener at localhost/46667] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-04-12 02:54:18,059 DEBUG [Listener at localhost/46667] store.ProcedureTree(114): Procedure Procedure(pid=44, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[0, 1, 2, 7, 8, 9] 2023-04-12 02:54:18,059 DEBUG [Listener at localhost/46667] store.ProcedureTree(114): Procedure Procedure(pid=45, ppid=44, class=org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure) stack ids=[3, 6] 2023-04-12 02:54:18,059 DEBUG [Listener at localhost/46667] store.ProcedureTree(114): Procedure Procedure(pid=46, ppid=45, class=org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure) stack ids=[4, 5] 2023-04-12 02:54:18,060 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-04-12 02:54:18,060 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-04-12 02:54:18,060 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-04-12 02:54:18,060 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-04-12 02:54:18,061 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-12 02:54:18,061 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=cc9012ec0b656baa24d04368fbe026c4, daughterA=e2a9b8fe15a781798c0b49e48ff5aad7, daughterB=6f55b5d6a460bd8c7569dee22f049104 2023-04-12 02:54:18,061 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-12 02:54:18,062 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-12 02:54:18,062 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-04-12 02:54:18,062 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=28, state=SUCCESS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=315e13e868a7f68a3ce37802b2f6312d, daughterA=4e1092f3c00484f8f09c730861225d43, daughterB=291734c00d212fd203b04cf8872d4649 2023-04-12 02:54:18,063 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=35, state=SUCCESS; DisableTableProcedure table=testSplitWithoutPONR 2023-04-12 02:54:18,063 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=40, state=SUCCESS; DeleteTableProcedure table=testSplitWithoutPONR 2023-04-12 02:54:18,063 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=41, state=SUCCESS; CreateTableProcedure table=testRollbackAndDoubleExecution 2023-04-12 02:54:18,064 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(468): Loading pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_BEFORE_META; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=96cdae2f7638212730b694522b9a464a, daughterA=fdd18e9cf539c13d863267ce16031583, daughterB=578095173fbc35e181cce14464037c2e 2023-04-12 02:54:18,064 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(468): Loading pid=45, ppid=44, state=SUCCESS; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=96cdae2f7638212730b694522b9a464a, UNASSIGN 2023-04-12 02:54:18,064 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(468): Loading pid=46, ppid=45, state=SUCCESS; CloseRegionProcedure 96cdae2f7638212730b694522b9a464a, server=jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:18,064 DEBUG [Listener at localhost/46667] procedure2.Procedure(966): pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_BEFORE_META; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=96cdae2f7638212730b694522b9a464a, daughterA=fdd18e9cf539c13d863267ce16031583, daughterB=578095173fbc35e181cce14464037c2e held the lock before restarting, call acquireLock to restore it. 2023-04-12 02:54:18,064 INFO [Listener at localhost/46667] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_BEFORE_META; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=96cdae2f7638212730b694522b9a464a, daughterA=fdd18e9cf539c13d863267ce16031583, daughterB=578095173fbc35e181cce14464037c2e 2023-04-12 02:54:18,064 INFO [Listener at localhost/46667] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_BEFORE_META; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=96cdae2f7638212730b694522b9a464a, daughterA=fdd18e9cf539c13d863267ce16031583, daughterB=578095173fbc35e181cce14464037c2e 2023-04-12 02:54:18,065 INFO [Listener at localhost/46667] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_BEFORE_META; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=96cdae2f7638212730b694522b9a464a, daughterA=fdd18e9cf539c13d863267ce16031583, daughterB=578095173fbc35e181cce14464037c2e 2023-04-12 02:54:18,065 DEBUG [Listener at localhost/46667] procedure2.Procedure(946): pid=45, ppid=44, state=SUCCESS; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=96cdae2f7638212730b694522b9a464a, UNASSIGN didn't hold the lock before restarting, skip acquiring lock. 2023-04-12 02:54:18,065 DEBUG [Listener at localhost/46667] procedure2.Procedure(946): pid=46, ppid=45, state=SUCCESS; CloseRegionProcedure 96cdae2f7638212730b694522b9a464a, server=jenkins-hbase4.apache.org,44679,1681268031772 didn't hold the lock before restarting, skip acquiring lock. 2023-04-12 02:54:18,065 DEBUG [Listener at localhost/46667] procedure2.StateMachineProcedure(225): Abort requested for pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_BEFORE_META; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=96cdae2f7638212730b694522b9a464a, daughterA=fdd18e9cf539c13d863267ce16031583, daughterB=578095173fbc35e181cce14464037c2e 2023-04-12 02:54:18,065 INFO [Listener at localhost/46667] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 6 msec 2023-04-12 02:54:18,066 DEBUG [Listener at localhost/46667] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-04-12 02:54:18,066 DEBUG [Listener at localhost/46667] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase4.apache.org,44679,1681268031772, table=hbase:meta, region=1588230740 2023-04-12 02:54:18,067 INFO [Listener at localhost/46667] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1681268088067 2023-04-12 02:54:18,067 DEBUG [Listener at localhost/46667] assignment.AssignmentManager(1517): Joining cluster... 2023-04-12 02:54:18,069 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_BEFORE_META; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=96cdae2f7638212730b694522b9a464a, daughterA=fdd18e9cf539c13d863267ce16031583, daughterB=578095173fbc35e181cce14464037c2e 2023-04-12 02:54:18,071 DEBUG [Listener at localhost/46667] assignment.RegionStateStore(147): Load hbase:meta entry region=1f255e980a31bb266f2206a30aa2ae3f, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,44679,1681268031772, regionLocation=jenkins-hbase4.apache.org,44679,1681268031772, openSeqNum=2 2023-04-12 02:54:18,071 DEBUG [Listener at localhost/46667] assignment.RegionStateStore(147): Load hbase:meta entry region=96cdae2f7638212730b694522b9a464a, regionState=CLOSED, lastHost=jenkins-hbase4.apache.org,44679,1681268031772, regionLocation=jenkins-hbase4.apache.org,44679,1681268031772, openSeqNum=2 2023-04-12 02:54:18,072 INFO [Listener at localhost/46667] assignment.AssignmentManager(1529): Number of RegionServers=3 2023-04-12 02:54:18,072 INFO [Listener at localhost/46667] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1681268118072 2023-04-12 02:54:18,072 INFO [Listener at localhost/46667] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1681268178072 2023-04-12 02:54:18,072 INFO [Listener at localhost/46667] assignment.AssignmentManager(1536): Joined the cluster in 5 msec 2023-04-12 02:54:18,072 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-04-12 02:54:18,108 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-04-12 02:54:18,108 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1581): TESTING: Kill before store update 2023-04-12 02:54:18,108 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-04-12 02:54:18,109 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill before store update at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeRollback(ProcedureExecutor.java:1583) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeRollback(ProcedureExecutor.java:1498) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1349) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-04-12 02:54:18,323 INFO [Listener at localhost/46667] procedure.MasterProcedureTestingUtility(509): Restart 1 rollback state: pid=44, state=FAILED:SPLIT_TABLE_REGION_WRITE_MAX_SEQUENCE_ID_FILE, locked=true, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=96cdae2f7638212730b694522b9a464a, daughterA=fdd18e9cf539c13d863267ce16031583, daughterB=578095173fbc35e181cce14464037c2e 2023-04-12 02:54:18,323 INFO [Listener at localhost/46667] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-04-12 02:54:18,323 INFO [Listener at localhost/46667] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-04-12 02:54:18,323 INFO [Listener at localhost/46667] assignment.AssignmentManager(315): Stopping assignment manager 2023-04-12 02:54:18,323 ERROR [Listener at localhost/46667] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-04-12 02:54:18,323 INFO [Listener at localhost/46667] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-04-12 02:54:18,324 INFO [Listener at localhost/46667] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-04-12 02:54:18,324 INFO [Listener at localhost/46667] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-04-12 02:54:18,324 INFO [Listener at localhost/46667] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-04-12 02:54:18,324 INFO [Listener at localhost/46667] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-04-12 02:54:18,325 DEBUG [Listener at localhost/46667] store.ProcedureTree(114): Procedure Procedure(pid=44, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[0, 1, 2, 7, 8, 9, 10] 2023-04-12 02:54:18,325 DEBUG [Listener at localhost/46667] store.ProcedureTree(114): Procedure Procedure(pid=45, ppid=44, class=org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure) stack ids=[3, 6] 2023-04-12 02:54:18,325 DEBUG [Listener at localhost/46667] store.ProcedureTree(114): Procedure Procedure(pid=46, ppid=45, class=org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure) stack ids=[4, 5] 2023-04-12 02:54:18,326 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-04-12 02:54:18,326 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-04-12 02:54:18,326 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-04-12 02:54:18,326 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-04-12 02:54:18,327 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-12 02:54:18,327 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=cc9012ec0b656baa24d04368fbe026c4, daughterA=e2a9b8fe15a781798c0b49e48ff5aad7, daughterB=6f55b5d6a460bd8c7569dee22f049104 2023-04-12 02:54:18,327 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-12 02:54:18,327 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-12 02:54:18,327 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-04-12 02:54:18,328 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=28, state=SUCCESS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=315e13e868a7f68a3ce37802b2f6312d, daughterA=4e1092f3c00484f8f09c730861225d43, daughterB=291734c00d212fd203b04cf8872d4649 2023-04-12 02:54:18,328 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=35, state=SUCCESS; DisableTableProcedure table=testSplitWithoutPONR 2023-04-12 02:54:18,328 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=40, state=SUCCESS; DeleteTableProcedure table=testSplitWithoutPONR 2023-04-12 02:54:18,329 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=41, state=SUCCESS; CreateTableProcedure table=testRollbackAndDoubleExecution 2023-04-12 02:54:18,329 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(468): Loading pid=44, state=FAILED:SPLIT_TABLE_REGION_PRE_OPERATION_BEFORE_META, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=96cdae2f7638212730b694522b9a464a, daughterA=fdd18e9cf539c13d863267ce16031583, daughterB=578095173fbc35e181cce14464037c2e 2023-04-12 02:54:18,329 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(468): Loading pid=45, ppid=44, state=SUCCESS; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=96cdae2f7638212730b694522b9a464a, UNASSIGN 2023-04-12 02:54:18,329 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(468): Loading pid=46, ppid=45, state=SUCCESS; CloseRegionProcedure 96cdae2f7638212730b694522b9a464a, server=jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:18,330 DEBUG [Listener at localhost/46667] procedure2.Procedure(966): pid=44, state=FAILED:SPLIT_TABLE_REGION_PRE_OPERATION_BEFORE_META, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=96cdae2f7638212730b694522b9a464a, daughterA=fdd18e9cf539c13d863267ce16031583, daughterB=578095173fbc35e181cce14464037c2e held the lock before restarting, call acquireLock to restore it. 2023-04-12 02:54:18,330 INFO [Listener at localhost/46667] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=FAILED:SPLIT_TABLE_REGION_PRE_OPERATION_BEFORE_META, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=96cdae2f7638212730b694522b9a464a, daughterA=fdd18e9cf539c13d863267ce16031583, daughterB=578095173fbc35e181cce14464037c2e 2023-04-12 02:54:18,330 INFO [Listener at localhost/46667] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=FAILED:SPLIT_TABLE_REGION_PRE_OPERATION_BEFORE_META, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=96cdae2f7638212730b694522b9a464a, daughterA=fdd18e9cf539c13d863267ce16031583, daughterB=578095173fbc35e181cce14464037c2e 2023-04-12 02:54:18,330 INFO [Listener at localhost/46667] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=FAILED:SPLIT_TABLE_REGION_PRE_OPERATION_BEFORE_META, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=96cdae2f7638212730b694522b9a464a, daughterA=fdd18e9cf539c13d863267ce16031583, daughterB=578095173fbc35e181cce14464037c2e 2023-04-12 02:54:18,330 DEBUG [Listener at localhost/46667] procedure2.Procedure(946): pid=45, ppid=44, state=SUCCESS; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=96cdae2f7638212730b694522b9a464a, UNASSIGN didn't hold the lock before restarting, skip acquiring lock. 2023-04-12 02:54:18,330 DEBUG [Listener at localhost/46667] procedure2.Procedure(946): pid=46, ppid=45, state=SUCCESS; CloseRegionProcedure 96cdae2f7638212730b694522b9a464a, server=jenkins-hbase4.apache.org,44679,1681268031772 didn't hold the lock before restarting, skip acquiring lock. 2023-04-12 02:54:18,330 INFO [Listener at localhost/46667] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 5 msec 2023-04-12 02:54:18,331 DEBUG [Listener at localhost/46667] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-04-12 02:54:18,331 DEBUG [Listener at localhost/46667] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase4.apache.org,44679,1681268031772, table=hbase:meta, region=1588230740 2023-04-12 02:54:18,331 INFO [Listener at localhost/46667] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1681268088331 2023-04-12 02:54:18,334 INFO [PEWorker-1] procedure2.ProcedureExecutor(1357): LOCK_EVENT_WAIT rollback...pid=44, state=FAILED:SPLIT_TABLE_REGION_PRE_OPERATION_BEFORE_META, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=96cdae2f7638212730b694522b9a464a, daughterA=fdd18e9cf539c13d863267ce16031583, daughterB=578095173fbc35e181cce14464037c2e 2023-04-12 02:54:18,334 DEBUG [Listener at localhost/46667] assignment.AssignmentManager(1517): Joining cluster... 2023-04-12 02:54:18,336 DEBUG [Listener at localhost/46667] assignment.RegionStateStore(147): Load hbase:meta entry region=1f255e980a31bb266f2206a30aa2ae3f, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,44679,1681268031772, regionLocation=jenkins-hbase4.apache.org,44679,1681268031772, openSeqNum=2 2023-04-12 02:54:18,336 DEBUG [Listener at localhost/46667] assignment.RegionStateStore(147): Load hbase:meta entry region=96cdae2f7638212730b694522b9a464a, regionState=CLOSED, lastHost=jenkins-hbase4.apache.org,44679,1681268031772, regionLocation=jenkins-hbase4.apache.org,44679,1681268031772, openSeqNum=2 2023-04-12 02:54:18,337 INFO [Listener at localhost/46667] assignment.AssignmentManager(1529): Number of RegionServers=3 2023-04-12 02:54:18,337 INFO [Listener at localhost/46667] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1681268118337 2023-04-12 02:54:18,337 INFO [Listener at localhost/46667] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1681268178337 2023-04-12 02:54:18,337 INFO [Listener at localhost/46667] assignment.AssignmentManager(1536): Joined the cluster in 2 msec 2023-04-12 02:54:18,337 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-04-12 02:54:18,339 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-04-12 02:54:18,339 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1581): TESTING: Kill before store update 2023-04-12 02:54:18,339 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-04-12 02:54:18,339 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill before store update at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeRollback(ProcedureExecutor.java:1583) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeRollback(ProcedureExecutor.java:1498) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1349) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-04-12 02:54:18,588 INFO [Listener at localhost/46667] procedure.MasterProcedureTestingUtility(509): Restart 2 rollback state: pid=44, state=FAILED:SPLIT_TABLE_REGION_CREATE_DAUGHTER_REGIONS, locked=true, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=96cdae2f7638212730b694522b9a464a, daughterA=fdd18e9cf539c13d863267ce16031583, daughterB=578095173fbc35e181cce14464037c2e 2023-04-12 02:54:18,588 INFO [Listener at localhost/46667] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-04-12 02:54:18,588 INFO [Listener at localhost/46667] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-04-12 02:54:18,588 INFO [Listener at localhost/46667] assignment.AssignmentManager(315): Stopping assignment manager 2023-04-12 02:54:18,588 ERROR [Listener at localhost/46667] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-04-12 02:54:18,588 INFO [Listener at localhost/46667] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-04-12 02:54:18,588 INFO [Listener at localhost/46667] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-04-12 02:54:18,588 INFO [Listener at localhost/46667] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-04-12 02:54:18,589 INFO [Listener at localhost/46667] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-04-12 02:54:18,589 INFO [Listener at localhost/46667] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-04-12 02:54:18,590 DEBUG [Listener at localhost/46667] store.ProcedureTree(114): Procedure Procedure(pid=44, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[0, 1, 2, 7, 8, 9] 2023-04-12 02:54:18,590 DEBUG [Listener at localhost/46667] store.ProcedureTree(114): Procedure Procedure(pid=45, ppid=44, class=org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure) stack ids=[3, 6] 2023-04-12 02:54:18,590 DEBUG [Listener at localhost/46667] store.ProcedureTree(114): Procedure Procedure(pid=46, ppid=45, class=org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure) stack ids=[4, 5] 2023-04-12 02:54:18,590 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-04-12 02:54:18,591 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-04-12 02:54:18,591 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-04-12 02:54:18,591 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-04-12 02:54:18,591 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-12 02:54:18,592 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=cc9012ec0b656baa24d04368fbe026c4, daughterA=e2a9b8fe15a781798c0b49e48ff5aad7, daughterB=6f55b5d6a460bd8c7569dee22f049104 2023-04-12 02:54:18,592 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-12 02:54:18,592 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-12 02:54:18,592 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-04-12 02:54:18,593 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=28, state=SUCCESS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=315e13e868a7f68a3ce37802b2f6312d, daughterA=4e1092f3c00484f8f09c730861225d43, daughterB=291734c00d212fd203b04cf8872d4649 2023-04-12 02:54:18,593 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=35, state=SUCCESS; DisableTableProcedure table=testSplitWithoutPONR 2023-04-12 02:54:18,593 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=40, state=SUCCESS; DeleteTableProcedure table=testSplitWithoutPONR 2023-04-12 02:54:18,593 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=41, state=SUCCESS; CreateTableProcedure table=testRollbackAndDoubleExecution 2023-04-12 02:54:18,594 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(468): Loading pid=44, state=FAILED:SPLIT_TABLE_REGION_WRITE_MAX_SEQUENCE_ID_FILE, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=96cdae2f7638212730b694522b9a464a, daughterA=fdd18e9cf539c13d863267ce16031583, daughterB=578095173fbc35e181cce14464037c2e 2023-04-12 02:54:18,594 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(468): Loading pid=45, ppid=44, state=SUCCESS; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=96cdae2f7638212730b694522b9a464a, UNASSIGN 2023-04-12 02:54:18,594 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(468): Loading pid=46, ppid=45, state=SUCCESS; CloseRegionProcedure 96cdae2f7638212730b694522b9a464a, server=jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:18,594 DEBUG [Listener at localhost/46667] procedure2.Procedure(966): pid=44, state=FAILED:SPLIT_TABLE_REGION_WRITE_MAX_SEQUENCE_ID_FILE, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=96cdae2f7638212730b694522b9a464a, daughterA=fdd18e9cf539c13d863267ce16031583, daughterB=578095173fbc35e181cce14464037c2e held the lock before restarting, call acquireLock to restore it. 2023-04-12 02:54:18,594 INFO [Listener at localhost/46667] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=FAILED:SPLIT_TABLE_REGION_WRITE_MAX_SEQUENCE_ID_FILE, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=96cdae2f7638212730b694522b9a464a, daughterA=fdd18e9cf539c13d863267ce16031583, daughterB=578095173fbc35e181cce14464037c2e 2023-04-12 02:54:18,594 INFO [Listener at localhost/46667] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=FAILED:SPLIT_TABLE_REGION_WRITE_MAX_SEQUENCE_ID_FILE, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=96cdae2f7638212730b694522b9a464a, daughterA=fdd18e9cf539c13d863267ce16031583, daughterB=578095173fbc35e181cce14464037c2e 2023-04-12 02:54:18,594 INFO [Listener at localhost/46667] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=FAILED:SPLIT_TABLE_REGION_WRITE_MAX_SEQUENCE_ID_FILE, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=96cdae2f7638212730b694522b9a464a, daughterA=fdd18e9cf539c13d863267ce16031583, daughterB=578095173fbc35e181cce14464037c2e 2023-04-12 02:54:18,594 DEBUG [Listener at localhost/46667] procedure2.Procedure(946): pid=45, ppid=44, state=SUCCESS; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=96cdae2f7638212730b694522b9a464a, UNASSIGN didn't hold the lock before restarting, skip acquiring lock. 2023-04-12 02:54:18,594 DEBUG [Listener at localhost/46667] procedure2.Procedure(946): pid=46, ppid=45, state=SUCCESS; CloseRegionProcedure 96cdae2f7638212730b694522b9a464a, server=jenkins-hbase4.apache.org,44679,1681268031772 didn't hold the lock before restarting, skip acquiring lock. 2023-04-12 02:54:18,594 INFO [Listener at localhost/46667] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 5 msec 2023-04-12 02:54:18,595 DEBUG [Listener at localhost/46667] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-04-12 02:54:18,595 DEBUG [Listener at localhost/46667] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase4.apache.org,44679,1681268031772, table=hbase:meta, region=1588230740 2023-04-12 02:54:18,596 INFO [Listener at localhost/46667] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1681268088596 2023-04-12 02:54:18,597 DEBUG [Listener at localhost/46667] assignment.AssignmentManager(1517): Joining cluster... 2023-04-12 02:54:18,598 INFO [PEWorker-1] procedure2.ProcedureExecutor(1357): LOCK_EVENT_WAIT rollback...pid=44, state=FAILED:SPLIT_TABLE_REGION_WRITE_MAX_SEQUENCE_ID_FILE, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=96cdae2f7638212730b694522b9a464a, daughterA=fdd18e9cf539c13d863267ce16031583, daughterB=578095173fbc35e181cce14464037c2e 2023-04-12 02:54:18,599 DEBUG [Listener at localhost/46667] assignment.RegionStateStore(147): Load hbase:meta entry region=1f255e980a31bb266f2206a30aa2ae3f, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,44679,1681268031772, regionLocation=jenkins-hbase4.apache.org,44679,1681268031772, openSeqNum=2 2023-04-12 02:54:18,600 DEBUG [Listener at localhost/46667] assignment.RegionStateStore(147): Load hbase:meta entry region=96cdae2f7638212730b694522b9a464a, regionState=CLOSED, lastHost=jenkins-hbase4.apache.org,44679,1681268031772, regionLocation=jenkins-hbase4.apache.org,44679,1681268031772, openSeqNum=2 2023-04-12 02:54:18,600 INFO [Listener at localhost/46667] assignment.AssignmentManager(1529): Number of RegionServers=3 2023-04-12 02:54:18,600 INFO [Listener at localhost/46667] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1681268118600 2023-04-12 02:54:18,600 INFO [Listener at localhost/46667] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1681268178600 2023-04-12 02:54:18,600 INFO [Listener at localhost/46667] assignment.AssignmentManager(1536): Joined the cluster in 3 msec 2023-04-12 02:54:18,600 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-04-12 02:54:18,603 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-04-12 02:54:18,603 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1581): TESTING: Kill before store update 2023-04-12 02:54:18,603 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-04-12 02:54:18,603 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill before store update at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeRollback(ProcedureExecutor.java:1583) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeRollback(ProcedureExecutor.java:1498) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1349) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-04-12 02:54:18,850 INFO [Listener at localhost/46667] procedure.MasterProcedureTestingUtility(509): Restart 3 rollback state: pid=44, state=FAILED:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS, locked=true, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=96cdae2f7638212730b694522b9a464a, daughterA=fdd18e9cf539c13d863267ce16031583, daughterB=578095173fbc35e181cce14464037c2e 2023-04-12 02:54:18,850 INFO [Listener at localhost/46667] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-04-12 02:54:18,851 INFO [Listener at localhost/46667] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-04-12 02:54:18,851 INFO [Listener at localhost/46667] assignment.AssignmentManager(315): Stopping assignment manager 2023-04-12 02:54:18,851 ERROR [Listener at localhost/46667] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-04-12 02:54:18,851 INFO [Listener at localhost/46667] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-04-12 02:54:18,851 INFO [Listener at localhost/46667] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-04-12 02:54:18,851 INFO [Listener at localhost/46667] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-04-12 02:54:18,851 INFO [Listener at localhost/46667] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-04-12 02:54:18,852 INFO [Listener at localhost/46667] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-04-12 02:54:18,853 DEBUG [Listener at localhost/46667] store.ProcedureTree(114): Procedure Procedure(pid=44, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[0, 1, 2, 7, 8] 2023-04-12 02:54:18,853 DEBUG [Listener at localhost/46667] store.ProcedureTree(114): Procedure Procedure(pid=45, ppid=44, class=org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure) stack ids=[3, 6] 2023-04-12 02:54:18,853 DEBUG [Listener at localhost/46667] store.ProcedureTree(114): Procedure Procedure(pid=46, ppid=45, class=org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure) stack ids=[4, 5] 2023-04-12 02:54:18,853 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-04-12 02:54:18,853 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-04-12 02:54:18,854 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-04-12 02:54:18,854 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-04-12 02:54:18,854 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-12 02:54:18,854 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=cc9012ec0b656baa24d04368fbe026c4, daughterA=e2a9b8fe15a781798c0b49e48ff5aad7, daughterB=6f55b5d6a460bd8c7569dee22f049104 2023-04-12 02:54:18,854 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-12 02:54:18,854 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-12 02:54:18,855 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-04-12 02:54:18,855 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=28, state=SUCCESS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=315e13e868a7f68a3ce37802b2f6312d, daughterA=4e1092f3c00484f8f09c730861225d43, daughterB=291734c00d212fd203b04cf8872d4649 2023-04-12 02:54:18,855 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=35, state=SUCCESS; DisableTableProcedure table=testSplitWithoutPONR 2023-04-12 02:54:18,855 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=40, state=SUCCESS; DeleteTableProcedure table=testSplitWithoutPONR 2023-04-12 02:54:18,855 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=41, state=SUCCESS; CreateTableProcedure table=testRollbackAndDoubleExecution 2023-04-12 02:54:18,856 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(468): Loading pid=44, state=FAILED:SPLIT_TABLE_REGION_CREATE_DAUGHTER_REGIONS, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=96cdae2f7638212730b694522b9a464a, daughterA=fdd18e9cf539c13d863267ce16031583, daughterB=578095173fbc35e181cce14464037c2e 2023-04-12 02:54:18,856 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(468): Loading pid=45, ppid=44, state=SUCCESS; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=96cdae2f7638212730b694522b9a464a, UNASSIGN 2023-04-12 02:54:18,856 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(468): Loading pid=46, ppid=45, state=SUCCESS; CloseRegionProcedure 96cdae2f7638212730b694522b9a464a, server=jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:18,856 DEBUG [Listener at localhost/46667] procedure2.Procedure(966): pid=44, state=FAILED:SPLIT_TABLE_REGION_CREATE_DAUGHTER_REGIONS, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=96cdae2f7638212730b694522b9a464a, daughterA=fdd18e9cf539c13d863267ce16031583, daughterB=578095173fbc35e181cce14464037c2e held the lock before restarting, call acquireLock to restore it. 2023-04-12 02:54:18,856 INFO [Listener at localhost/46667] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=FAILED:SPLIT_TABLE_REGION_CREATE_DAUGHTER_REGIONS, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=96cdae2f7638212730b694522b9a464a, daughterA=fdd18e9cf539c13d863267ce16031583, daughterB=578095173fbc35e181cce14464037c2e 2023-04-12 02:54:18,856 INFO [Listener at localhost/46667] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=FAILED:SPLIT_TABLE_REGION_CREATE_DAUGHTER_REGIONS, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=96cdae2f7638212730b694522b9a464a, daughterA=fdd18e9cf539c13d863267ce16031583, daughterB=578095173fbc35e181cce14464037c2e 2023-04-12 02:54:18,856 INFO [Listener at localhost/46667] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=FAILED:SPLIT_TABLE_REGION_CREATE_DAUGHTER_REGIONS, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=96cdae2f7638212730b694522b9a464a, daughterA=fdd18e9cf539c13d863267ce16031583, daughterB=578095173fbc35e181cce14464037c2e 2023-04-12 02:54:18,856 DEBUG [Listener at localhost/46667] procedure2.Procedure(946): pid=45, ppid=44, state=SUCCESS; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=96cdae2f7638212730b694522b9a464a, UNASSIGN didn't hold the lock before restarting, skip acquiring lock. 2023-04-12 02:54:18,857 DEBUG [Listener at localhost/46667] procedure2.Procedure(946): pid=46, ppid=45, state=SUCCESS; CloseRegionProcedure 96cdae2f7638212730b694522b9a464a, server=jenkins-hbase4.apache.org,44679,1681268031772 didn't hold the lock before restarting, skip acquiring lock. 2023-04-12 02:54:18,857 INFO [Listener at localhost/46667] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 4 msec 2023-04-12 02:54:18,857 DEBUG [Listener at localhost/46667] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-04-12 02:54:18,858 DEBUG [Listener at localhost/46667] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase4.apache.org,44679,1681268031772, table=hbase:meta, region=1588230740 2023-04-12 02:54:18,858 INFO [Listener at localhost/46667] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1681268088858 2023-04-12 02:54:18,859 DEBUG [Listener at localhost/46667] assignment.AssignmentManager(1517): Joining cluster... 2023-04-12 02:54:18,860 INFO [PEWorker-1] procedure2.ProcedureExecutor(1357): LOCK_EVENT_WAIT rollback...pid=44, state=FAILED:SPLIT_TABLE_REGION_CREATE_DAUGHTER_REGIONS, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=96cdae2f7638212730b694522b9a464a, daughterA=fdd18e9cf539c13d863267ce16031583, daughterB=578095173fbc35e181cce14464037c2e 2023-04-12 02:54:18,862 DEBUG [Listener at localhost/46667] assignment.RegionStateStore(147): Load hbase:meta entry region=1f255e980a31bb266f2206a30aa2ae3f, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,44679,1681268031772, regionLocation=jenkins-hbase4.apache.org,44679,1681268031772, openSeqNum=2 2023-04-12 02:54:18,862 DEBUG [Listener at localhost/46667] assignment.RegionStateStore(147): Load hbase:meta entry region=96cdae2f7638212730b694522b9a464a, regionState=CLOSED, lastHost=jenkins-hbase4.apache.org,44679,1681268031772, regionLocation=jenkins-hbase4.apache.org,44679,1681268031772, openSeqNum=2 2023-04-12 02:54:18,862 INFO [Listener at localhost/46667] assignment.AssignmentManager(1529): Number of RegionServers=3 2023-04-12 02:54:18,862 INFO [Listener at localhost/46667] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1681268118862 2023-04-12 02:54:18,862 INFO [Listener at localhost/46667] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1681268178862 2023-04-12 02:54:18,862 INFO [Listener at localhost/46667] assignment.AssignmentManager(1536): Joined the cluster in 3 msec 2023-04-12 02:54:18,862 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-04-12 02:54:18,866 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1054): Stored [pid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=96cdae2f7638212730b694522b9a464a, ASSIGN] 2023-04-12 02:54:18,866 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-04-12 02:54:18,866 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1581): TESTING: Kill before store update 2023-04-12 02:54:18,866 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-04-12 02:54:18,866 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill before store update at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeRollback(ProcedureExecutor.java:1583) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeRollback(ProcedureExecutor.java:1498) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1349) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-04-12 02:54:19,112 INFO [Listener at localhost/46667] procedure.MasterProcedureTestingUtility(509): Restart 4 rollback state: pid=44, state=FAILED:SPLIT_TABLE_REGION_CLOSE_PARENT_REGION, locked=true, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=96cdae2f7638212730b694522b9a464a, daughterA=fdd18e9cf539c13d863267ce16031583, daughterB=578095173fbc35e181cce14464037c2e 2023-04-12 02:54:19,113 INFO [Listener at localhost/46667] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-04-12 02:54:19,113 INFO [Listener at localhost/46667] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-04-12 02:54:19,113 INFO [Listener at localhost/46667] assignment.AssignmentManager(315): Stopping assignment manager 2023-04-12 02:54:19,113 ERROR [Listener at localhost/46667] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-04-12 02:54:19,113 INFO [Listener at localhost/46667] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-04-12 02:54:19,113 INFO [Listener at localhost/46667] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-04-12 02:54:19,113 INFO [Listener at localhost/46667] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-04-12 02:54:19,113 INFO [Listener at localhost/46667] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-04-12 02:54:19,114 INFO [Listener at localhost/46667] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-04-12 02:54:19,115 DEBUG [Listener at localhost/46667] store.ProcedureTree(114): Procedure Procedure(pid=44, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[0, 1, 2, 7] 2023-04-12 02:54:19,115 DEBUG [Listener at localhost/46667] store.ProcedureTree(114): Procedure Procedure(pid=45, ppid=44, class=org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure) stack ids=[3, 6] 2023-04-12 02:54:19,115 DEBUG [Listener at localhost/46667] store.ProcedureTree(114): Procedure Procedure(pid=46, ppid=45, class=org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure) stack ids=[4, 5] 2023-04-12 02:54:19,115 DEBUG [Listener at localhost/46667] store.ProcedureTree(114): Procedure Procedure(pid=47, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure) stack ids=[] 2023-04-12 02:54:19,115 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-04-12 02:54:19,116 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-04-12 02:54:19,116 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-04-12 02:54:19,116 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-04-12 02:54:19,116 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-12 02:54:19,117 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=cc9012ec0b656baa24d04368fbe026c4, daughterA=e2a9b8fe15a781798c0b49e48ff5aad7, daughterB=6f55b5d6a460bd8c7569dee22f049104 2023-04-12 02:54:19,117 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-12 02:54:19,117 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-12 02:54:19,117 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-04-12 02:54:19,117 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=28, state=SUCCESS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=315e13e868a7f68a3ce37802b2f6312d, daughterA=4e1092f3c00484f8f09c730861225d43, daughterB=291734c00d212fd203b04cf8872d4649 2023-04-12 02:54:19,118 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=35, state=SUCCESS; DisableTableProcedure table=testSplitWithoutPONR 2023-04-12 02:54:19,118 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=40, state=SUCCESS; DeleteTableProcedure table=testSplitWithoutPONR 2023-04-12 02:54:19,118 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=41, state=SUCCESS; CreateTableProcedure table=testRollbackAndDoubleExecution 2023-04-12 02:54:19,119 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(468): Loading pid=44, state=FAILED:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=96cdae2f7638212730b694522b9a464a, daughterA=fdd18e9cf539c13d863267ce16031583, daughterB=578095173fbc35e181cce14464037c2e 2023-04-12 02:54:19,119 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(468): Loading pid=45, ppid=44, state=SUCCESS; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=96cdae2f7638212730b694522b9a464a, UNASSIGN 2023-04-12 02:54:19,119 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(468): Loading pid=46, ppid=45, state=SUCCESS; CloseRegionProcedure 96cdae2f7638212730b694522b9a464a, server=jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:19,119 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(468): Loading pid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=96cdae2f7638212730b694522b9a464a, ASSIGN 2023-04-12 02:54:19,119 DEBUG [Listener at localhost/46667] procedure2.Procedure(966): pid=44, state=FAILED:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=96cdae2f7638212730b694522b9a464a, daughterA=fdd18e9cf539c13d863267ce16031583, daughterB=578095173fbc35e181cce14464037c2e held the lock before restarting, call acquireLock to restore it. 2023-04-12 02:54:19,120 INFO [Listener at localhost/46667] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=FAILED:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=96cdae2f7638212730b694522b9a464a, daughterA=fdd18e9cf539c13d863267ce16031583, daughterB=578095173fbc35e181cce14464037c2e 2023-04-12 02:54:19,120 INFO [Listener at localhost/46667] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=FAILED:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=96cdae2f7638212730b694522b9a464a, daughterA=fdd18e9cf539c13d863267ce16031583, daughterB=578095173fbc35e181cce14464037c2e 2023-04-12 02:54:19,120 INFO [Listener at localhost/46667] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=FAILED:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=96cdae2f7638212730b694522b9a464a, daughterA=fdd18e9cf539c13d863267ce16031583, daughterB=578095173fbc35e181cce14464037c2e 2023-04-12 02:54:19,120 DEBUG [Listener at localhost/46667] procedure2.Procedure(946): pid=45, ppid=44, state=SUCCESS; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=96cdae2f7638212730b694522b9a464a, UNASSIGN didn't hold the lock before restarting, skip acquiring lock. 2023-04-12 02:54:19,120 DEBUG [Listener at localhost/46667] procedure2.Procedure(946): pid=46, ppid=45, state=SUCCESS; CloseRegionProcedure 96cdae2f7638212730b694522b9a464a, server=jenkins-hbase4.apache.org,44679,1681268031772 didn't hold the lock before restarting, skip acquiring lock. 2023-04-12 02:54:19,120 DEBUG [Listener at localhost/46667] procedure2.Procedure(946): pid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=96cdae2f7638212730b694522b9a464a, ASSIGN didn't hold the lock before restarting, skip acquiring lock. 2023-04-12 02:54:19,120 INFO [Listener at localhost/46667] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 6 msec 2023-04-12 02:54:19,121 DEBUG [Listener at localhost/46667] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-04-12 02:54:19,121 DEBUG [Listener at localhost/46667] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase4.apache.org,44679,1681268031772, table=hbase:meta, region=1588230740 2023-04-12 02:54:19,121 INFO [Listener at localhost/46667] assignment.AssignmentManager(305): Attach pid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=96cdae2f7638212730b694522b9a464a, ASSIGN to state=OFFLINE, location=null, table=testRollbackAndDoubleExecution, region=96cdae2f7638212730b694522b9a464a to restore RIT 2023-04-12 02:54:19,122 INFO [Listener at localhost/46667] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1681268089122 2023-04-12 02:54:19,124 DEBUG [Listener at localhost/46667] assignment.AssignmentManager(1517): Joining cluster... 2023-04-12 02:54:19,124 INFO [PEWorker-1] procedure2.ProcedureExecutor(1357): LOCK_EVENT_WAIT rollback...pid=44, state=FAILED:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=96cdae2f7638212730b694522b9a464a, daughterA=fdd18e9cf539c13d863267ce16031583, daughterB=578095173fbc35e181cce14464037c2e 2023-04-12 02:54:19,125 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=96cdae2f7638212730b694522b9a464a, ASSIGN 2023-04-12 02:54:19,127 DEBUG [Listener at localhost/46667] assignment.RegionStateStore(147): Load hbase:meta entry region=1f255e980a31bb266f2206a30aa2ae3f, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,44679,1681268031772, regionLocation=jenkins-hbase4.apache.org,44679,1681268031772, openSeqNum=2 2023-04-12 02:54:19,128 DEBUG [Listener at localhost/46667] assignment.RegionStateStore(147): Load hbase:meta entry region=96cdae2f7638212730b694522b9a464a, regionState=CLOSED, lastHost=jenkins-hbase4.apache.org,44679,1681268031772, regionLocation=jenkins-hbase4.apache.org,44679,1681268031772, openSeqNum=2 2023-04-12 02:54:19,128 INFO [Listener at localhost/46667] assignment.AssignmentManager(1529): Number of RegionServers=3 2023-04-12 02:54:19,128 INFO [Listener at localhost/46667] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1681268119128 2023-04-12 02:54:19,128 INFO [Listener at localhost/46667] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1681268179128 2023-04-12 02:54:19,128 INFO [Listener at localhost/46667] assignment.AssignmentManager(1536): Joined the cluster in 4 msec 2023-04-12 02:54:19,128 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-04-12 02:54:19,134 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-04-12 02:54:19,134 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1581): TESTING: Kill before store update 2023-04-12 02:54:19,134 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-04-12 02:54:19,134 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill before store update at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeRollback(ProcedureExecutor.java:1583) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeRollback(ProcedureExecutor.java:1498) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1349) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-04-12 02:54:19,378 INFO [Listener at localhost/46667] procedure.MasterProcedureTestingUtility(509): Restart 5 rollback state: pid=44, state=FAILED:SPLIT_TABLE_REGION_PRE_OPERATION, locked=true, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=96cdae2f7638212730b694522b9a464a, daughterA=fdd18e9cf539c13d863267ce16031583, daughterB=578095173fbc35e181cce14464037c2e 2023-04-12 02:54:19,378 INFO [Listener at localhost/46667] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-04-12 02:54:19,378 INFO [Listener at localhost/46667] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-04-12 02:54:19,378 INFO [Listener at localhost/46667] assignment.AssignmentManager(315): Stopping assignment manager 2023-04-12 02:54:19,381 ERROR [Listener at localhost/46667] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-04-12 02:54:19,381 INFO [Listener at localhost/46667] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-04-12 02:54:19,381 INFO [Listener at localhost/46667] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-04-12 02:54:19,381 INFO [Listener at localhost/46667] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-04-12 02:54:19,381 INFO [Listener at localhost/46667] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-04-12 02:54:19,382 INFO [Listener at localhost/46667] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-04-12 02:54:19,383 DEBUG [Listener at localhost/46667] store.ProcedureTree(114): Procedure Procedure(pid=44, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[0, 1, 2] 2023-04-12 02:54:19,383 DEBUG [Listener at localhost/46667] store.ProcedureTree(114): Procedure Procedure(pid=47, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure) stack ids=[] 2023-04-12 02:54:19,383 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-04-12 02:54:19,383 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-04-12 02:54:19,383 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-04-12 02:54:19,384 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-04-12 02:54:19,384 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-12 02:54:19,384 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=cc9012ec0b656baa24d04368fbe026c4, daughterA=e2a9b8fe15a781798c0b49e48ff5aad7, daughterB=6f55b5d6a460bd8c7569dee22f049104 2023-04-12 02:54:19,384 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-12 02:54:19,384 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-12 02:54:19,385 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-04-12 02:54:19,385 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=28, state=SUCCESS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=315e13e868a7f68a3ce37802b2f6312d, daughterA=4e1092f3c00484f8f09c730861225d43, daughterB=291734c00d212fd203b04cf8872d4649 2023-04-12 02:54:19,385 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=35, state=SUCCESS; DisableTableProcedure table=testSplitWithoutPONR 2023-04-12 02:54:19,385 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=40, state=SUCCESS; DeleteTableProcedure table=testSplitWithoutPONR 2023-04-12 02:54:19,386 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=41, state=SUCCESS; CreateTableProcedure table=testRollbackAndDoubleExecution 2023-04-12 02:54:19,386 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(468): Loading pid=44, state=FAILED:SPLIT_TABLE_REGION_CLOSE_PARENT_REGION, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=96cdae2f7638212730b694522b9a464a, daughterA=fdd18e9cf539c13d863267ce16031583, daughterB=578095173fbc35e181cce14464037c2e 2023-04-12 02:54:19,386 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(468): Loading pid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=96cdae2f7638212730b694522b9a464a, ASSIGN 2023-04-12 02:54:19,386 DEBUG [Listener at localhost/46667] procedure2.Procedure(966): pid=44, state=FAILED:SPLIT_TABLE_REGION_CLOSE_PARENT_REGION, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=96cdae2f7638212730b694522b9a464a, daughterA=fdd18e9cf539c13d863267ce16031583, daughterB=578095173fbc35e181cce14464037c2e held the lock before restarting, call acquireLock to restore it. 2023-04-12 02:54:19,386 INFO [Listener at localhost/46667] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=FAILED:SPLIT_TABLE_REGION_CLOSE_PARENT_REGION, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=96cdae2f7638212730b694522b9a464a, daughterA=fdd18e9cf539c13d863267ce16031583, daughterB=578095173fbc35e181cce14464037c2e 2023-04-12 02:54:19,386 INFO [Listener at localhost/46667] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=FAILED:SPLIT_TABLE_REGION_CLOSE_PARENT_REGION, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=96cdae2f7638212730b694522b9a464a, daughterA=fdd18e9cf539c13d863267ce16031583, daughterB=578095173fbc35e181cce14464037c2e 2023-04-12 02:54:19,386 INFO [Listener at localhost/46667] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=FAILED:SPLIT_TABLE_REGION_CLOSE_PARENT_REGION, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=96cdae2f7638212730b694522b9a464a, daughterA=fdd18e9cf539c13d863267ce16031583, daughterB=578095173fbc35e181cce14464037c2e 2023-04-12 02:54:19,386 DEBUG [Listener at localhost/46667] procedure2.Procedure(946): pid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=96cdae2f7638212730b694522b9a464a, ASSIGN didn't hold the lock before restarting, skip acquiring lock. 2023-04-12 02:54:19,387 INFO [Listener at localhost/46667] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 4 msec 2023-04-12 02:54:19,387 DEBUG [Listener at localhost/46667] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-04-12 02:54:19,388 DEBUG [Listener at localhost/46667] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase4.apache.org,44679,1681268031772, table=hbase:meta, region=1588230740 2023-04-12 02:54:19,388 INFO [Listener at localhost/46667] assignment.AssignmentManager(305): Attach pid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=96cdae2f7638212730b694522b9a464a, ASSIGN to state=OFFLINE, location=null, table=testRollbackAndDoubleExecution, region=96cdae2f7638212730b694522b9a464a to restore RIT 2023-04-12 02:54:19,388 INFO [Listener at localhost/46667] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1681268089388 2023-04-12 02:54:19,389 DEBUG [Listener at localhost/46667] assignment.AssignmentManager(1517): Joining cluster... 2023-04-12 02:54:19,390 INFO [PEWorker-1] procedure2.ProcedureExecutor(1357): LOCK_EVENT_WAIT rollback...pid=44, state=FAILED:SPLIT_TABLE_REGION_CLOSE_PARENT_REGION, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=96cdae2f7638212730b694522b9a464a, daughterA=fdd18e9cf539c13d863267ce16031583, daughterB=578095173fbc35e181cce14464037c2e 2023-04-12 02:54:19,391 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=96cdae2f7638212730b694522b9a464a, ASSIGN 2023-04-12 02:54:19,392 DEBUG [Listener at localhost/46667] assignment.RegionStateStore(147): Load hbase:meta entry region=1f255e980a31bb266f2206a30aa2ae3f, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,44679,1681268031772, regionLocation=jenkins-hbase4.apache.org,44679,1681268031772, openSeqNum=2 2023-04-12 02:54:19,393 DEBUG [Listener at localhost/46667] assignment.RegionStateStore(147): Load hbase:meta entry region=96cdae2f7638212730b694522b9a464a, regionState=CLOSED, lastHost=jenkins-hbase4.apache.org,44679,1681268031772, regionLocation=jenkins-hbase4.apache.org,44679,1681268031772, openSeqNum=2 2023-04-12 02:54:19,393 INFO [Listener at localhost/46667] assignment.AssignmentManager(1529): Number of RegionServers=3 2023-04-12 02:54:19,393 INFO [Listener at localhost/46667] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1681268119393 2023-04-12 02:54:19,393 INFO [Listener at localhost/46667] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1681268179393 2023-04-12 02:54:19,393 INFO [Listener at localhost/46667] assignment.AssignmentManager(1536): Joined the cluster in 3 msec 2023-04-12 02:54:19,393 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-04-12 02:54:19,395 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-04-12 02:54:19,395 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1581): TESTING: Kill before store update 2023-04-12 02:54:19,395 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-04-12 02:54:19,395 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill before store update at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeRollback(ProcedureExecutor.java:1583) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeRollback(ProcedureExecutor.java:1498) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1349) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-04-12 02:54:19,643 INFO [Listener at localhost/46667] procedure.MasterProcedureTestingUtility(509): Restart 6 rollback state: pid=44, state=FAILED:SPLIT_TABLE_REGION_PREPARE, locked=true, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=96cdae2f7638212730b694522b9a464a, daughterA=fdd18e9cf539c13d863267ce16031583, daughterB=578095173fbc35e181cce14464037c2e 2023-04-12 02:54:19,643 INFO [Listener at localhost/46667] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-04-12 02:54:19,643 INFO [Listener at localhost/46667] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-04-12 02:54:19,643 INFO [Listener at localhost/46667] assignment.AssignmentManager(315): Stopping assignment manager 2023-04-12 02:54:19,644 ERROR [Listener at localhost/46667] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-04-12 02:54:19,644 INFO [Listener at localhost/46667] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-04-12 02:54:19,644 INFO [Listener at localhost/46667] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-04-12 02:54:19,644 INFO [Listener at localhost/46667] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-04-12 02:54:19,644 INFO [Listener at localhost/46667] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-04-12 02:54:19,644 INFO [Listener at localhost/46667] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-04-12 02:54:19,645 DEBUG [Listener at localhost/46667] store.ProcedureTree(114): Procedure Procedure(pid=44, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[0, 1] 2023-04-12 02:54:19,645 DEBUG [Listener at localhost/46667] store.ProcedureTree(114): Procedure Procedure(pid=47, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure) stack ids=[] 2023-04-12 02:54:19,646 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-04-12 02:54:19,646 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-04-12 02:54:19,646 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-04-12 02:54:19,646 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-04-12 02:54:19,646 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-12 02:54:19,647 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=cc9012ec0b656baa24d04368fbe026c4, daughterA=e2a9b8fe15a781798c0b49e48ff5aad7, daughterB=6f55b5d6a460bd8c7569dee22f049104 2023-04-12 02:54:19,647 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-12 02:54:19,647 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-12 02:54:19,647 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-04-12 02:54:19,647 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=28, state=SUCCESS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=315e13e868a7f68a3ce37802b2f6312d, daughterA=4e1092f3c00484f8f09c730861225d43, daughterB=291734c00d212fd203b04cf8872d4649 2023-04-12 02:54:19,648 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=35, state=SUCCESS; DisableTableProcedure table=testSplitWithoutPONR 2023-04-12 02:54:19,648 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=40, state=SUCCESS; DeleteTableProcedure table=testSplitWithoutPONR 2023-04-12 02:54:19,648 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=41, state=SUCCESS; CreateTableProcedure table=testRollbackAndDoubleExecution 2023-04-12 02:54:19,648 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(468): Loading pid=44, state=FAILED:SPLIT_TABLE_REGION_PRE_OPERATION, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=96cdae2f7638212730b694522b9a464a, daughterA=fdd18e9cf539c13d863267ce16031583, daughterB=578095173fbc35e181cce14464037c2e 2023-04-12 02:54:19,648 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(468): Loading pid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=96cdae2f7638212730b694522b9a464a, ASSIGN 2023-04-12 02:54:19,648 DEBUG [Listener at localhost/46667] procedure2.Procedure(966): pid=44, state=FAILED:SPLIT_TABLE_REGION_PRE_OPERATION, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=96cdae2f7638212730b694522b9a464a, daughterA=fdd18e9cf539c13d863267ce16031583, daughterB=578095173fbc35e181cce14464037c2e held the lock before restarting, call acquireLock to restore it. 2023-04-12 02:54:19,649 INFO [Listener at localhost/46667] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=FAILED:SPLIT_TABLE_REGION_PRE_OPERATION, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=96cdae2f7638212730b694522b9a464a, daughterA=fdd18e9cf539c13d863267ce16031583, daughterB=578095173fbc35e181cce14464037c2e 2023-04-12 02:54:19,649 INFO [Listener at localhost/46667] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=FAILED:SPLIT_TABLE_REGION_PRE_OPERATION, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=96cdae2f7638212730b694522b9a464a, daughterA=fdd18e9cf539c13d863267ce16031583, daughterB=578095173fbc35e181cce14464037c2e 2023-04-12 02:54:19,649 INFO [Listener at localhost/46667] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=FAILED:SPLIT_TABLE_REGION_PRE_OPERATION, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=96cdae2f7638212730b694522b9a464a, daughterA=fdd18e9cf539c13d863267ce16031583, daughterB=578095173fbc35e181cce14464037c2e 2023-04-12 02:54:19,649 DEBUG [Listener at localhost/46667] procedure2.Procedure(946): pid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=96cdae2f7638212730b694522b9a464a, ASSIGN didn't hold the lock before restarting, skip acquiring lock. 2023-04-12 02:54:19,649 INFO [Listener at localhost/46667] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 4 msec 2023-04-12 02:54:19,650 DEBUG [Listener at localhost/46667] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-04-12 02:54:19,650 DEBUG [Listener at localhost/46667] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase4.apache.org,44679,1681268031772, table=hbase:meta, region=1588230740 2023-04-12 02:54:19,650 INFO [Listener at localhost/46667] assignment.AssignmentManager(305): Attach pid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=96cdae2f7638212730b694522b9a464a, ASSIGN to state=OFFLINE, location=null, table=testRollbackAndDoubleExecution, region=96cdae2f7638212730b694522b9a464a to restore RIT 2023-04-12 02:54:19,651 INFO [Listener at localhost/46667] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1681268089651 2023-04-12 02:54:19,653 INFO [PEWorker-1] procedure2.ProcedureExecutor(1357): LOCK_EVENT_WAIT rollback...pid=44, state=FAILED:SPLIT_TABLE_REGION_PRE_OPERATION, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=96cdae2f7638212730b694522b9a464a, daughterA=fdd18e9cf539c13d863267ce16031583, daughterB=578095173fbc35e181cce14464037c2e 2023-04-12 02:54:19,653 DEBUG [Listener at localhost/46667] assignment.AssignmentManager(1517): Joining cluster... 2023-04-12 02:54:19,653 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=96cdae2f7638212730b694522b9a464a, ASSIGN 2023-04-12 02:54:19,655 DEBUG [Listener at localhost/46667] assignment.RegionStateStore(147): Load hbase:meta entry region=1f255e980a31bb266f2206a30aa2ae3f, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,44679,1681268031772, regionLocation=jenkins-hbase4.apache.org,44679,1681268031772, openSeqNum=2 2023-04-12 02:54:19,656 DEBUG [Listener at localhost/46667] assignment.RegionStateStore(147): Load hbase:meta entry region=96cdae2f7638212730b694522b9a464a, regionState=CLOSED, lastHost=jenkins-hbase4.apache.org,44679,1681268031772, regionLocation=jenkins-hbase4.apache.org,44679,1681268031772, openSeqNum=2 2023-04-12 02:54:19,656 INFO [Listener at localhost/46667] assignment.AssignmentManager(1529): Number of RegionServers=3 2023-04-12 02:54:19,656 INFO [Listener at localhost/46667] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1681268119656 2023-04-12 02:54:19,656 INFO [Listener at localhost/46667] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1681268179656 2023-04-12 02:54:19,656 INFO [Listener at localhost/46667] assignment.AssignmentManager(1536): Joined the cluster in 3 msec 2023-04-12 02:54:19,656 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-04-12 02:54:19,658 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-04-12 02:54:19,658 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1581): TESTING: Kill before store update 2023-04-12 02:54:19,658 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-04-12 02:54:19,658 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill before store update at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeRollback(ProcedureExecutor.java:1583) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeRollback(ProcedureExecutor.java:1498) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1349) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-04-12 02:54:19,906 INFO [Listener at localhost/46667] procedure.MasterProcedureTestingUtility(509): Restart 7 rollback state: pid=44, state=FAILED:SPLIT_TABLE_REGION_PREPARE, locked=true, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=96cdae2f7638212730b694522b9a464a, daughterA=fdd18e9cf539c13d863267ce16031583, daughterB=578095173fbc35e181cce14464037c2e 2023-04-12 02:54:19,906 INFO [Listener at localhost/46667] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-04-12 02:54:19,906 INFO [Listener at localhost/46667] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-04-12 02:54:19,906 INFO [Listener at localhost/46667] assignment.AssignmentManager(315): Stopping assignment manager 2023-04-12 02:54:19,907 ERROR [Listener at localhost/46667] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-04-12 02:54:19,907 INFO [Listener at localhost/46667] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-04-12 02:54:19,907 INFO [Listener at localhost/46667] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-04-12 02:54:19,907 INFO [Listener at localhost/46667] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-04-12 02:54:19,907 INFO [Listener at localhost/46667] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-04-12 02:54:19,908 INFO [Listener at localhost/46667] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-04-12 02:54:19,908 DEBUG [Listener at localhost/46667] store.ProcedureTree(114): Procedure Procedure(pid=44, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[0] 2023-04-12 02:54:19,908 DEBUG [Listener at localhost/46667] store.ProcedureTree(114): Procedure Procedure(pid=47, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure) stack ids=[] 2023-04-12 02:54:19,909 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-04-12 02:54:19,909 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-04-12 02:54:19,909 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-04-12 02:54:19,909 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-04-12 02:54:19,910 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-12 02:54:19,910 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=cc9012ec0b656baa24d04368fbe026c4, daughterA=e2a9b8fe15a781798c0b49e48ff5aad7, daughterB=6f55b5d6a460bd8c7569dee22f049104 2023-04-12 02:54:19,910 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-12 02:54:19,911 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-12 02:54:19,911 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-04-12 02:54:19,911 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=28, state=SUCCESS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=315e13e868a7f68a3ce37802b2f6312d, daughterA=4e1092f3c00484f8f09c730861225d43, daughterB=291734c00d212fd203b04cf8872d4649 2023-04-12 02:54:19,911 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=35, state=SUCCESS; DisableTableProcedure table=testSplitWithoutPONR 2023-04-12 02:54:19,912 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=40, state=SUCCESS; DeleteTableProcedure table=testSplitWithoutPONR 2023-04-12 02:54:19,912 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=41, state=SUCCESS; CreateTableProcedure table=testRollbackAndDoubleExecution 2023-04-12 02:54:19,912 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(468): Loading pid=44, state=FAILED:SPLIT_TABLE_REGION_PREPARE, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=96cdae2f7638212730b694522b9a464a, daughterA=fdd18e9cf539c13d863267ce16031583, daughterB=578095173fbc35e181cce14464037c2e 2023-04-12 02:54:19,913 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(468): Loading pid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=96cdae2f7638212730b694522b9a464a, ASSIGN 2023-04-12 02:54:19,913 DEBUG [Listener at localhost/46667] procedure2.Procedure(966): pid=44, state=FAILED:SPLIT_TABLE_REGION_PREPARE, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=96cdae2f7638212730b694522b9a464a, daughterA=fdd18e9cf539c13d863267ce16031583, daughterB=578095173fbc35e181cce14464037c2e held the lock before restarting, call acquireLock to restore it. 2023-04-12 02:54:19,913 INFO [Listener at localhost/46667] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=FAILED:SPLIT_TABLE_REGION_PREPARE, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=96cdae2f7638212730b694522b9a464a, daughterA=fdd18e9cf539c13d863267ce16031583, daughterB=578095173fbc35e181cce14464037c2e 2023-04-12 02:54:19,913 INFO [Listener at localhost/46667] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=FAILED:SPLIT_TABLE_REGION_PREPARE, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=96cdae2f7638212730b694522b9a464a, daughterA=fdd18e9cf539c13d863267ce16031583, daughterB=578095173fbc35e181cce14464037c2e 2023-04-12 02:54:19,913 INFO [Listener at localhost/46667] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=FAILED:SPLIT_TABLE_REGION_PREPARE, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=96cdae2f7638212730b694522b9a464a, daughterA=fdd18e9cf539c13d863267ce16031583, daughterB=578095173fbc35e181cce14464037c2e 2023-04-12 02:54:19,913 DEBUG [Listener at localhost/46667] procedure2.Procedure(946): pid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=96cdae2f7638212730b694522b9a464a, ASSIGN didn't hold the lock before restarting, skip acquiring lock. 2023-04-12 02:54:19,913 INFO [Listener at localhost/46667] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 5 msec 2023-04-12 02:54:19,914 DEBUG [Listener at localhost/46667] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-04-12 02:54:19,914 DEBUG [Listener at localhost/46667] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase4.apache.org,44679,1681268031772, table=hbase:meta, region=1588230740 2023-04-12 02:54:19,915 INFO [Listener at localhost/46667] assignment.AssignmentManager(305): Attach pid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=96cdae2f7638212730b694522b9a464a, ASSIGN to state=OFFLINE, location=null, table=testRollbackAndDoubleExecution, region=96cdae2f7638212730b694522b9a464a to restore RIT 2023-04-12 02:54:19,915 INFO [Listener at localhost/46667] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1681268089915 2023-04-12 02:54:19,917 DEBUG [Listener at localhost/46667] assignment.AssignmentManager(1517): Joining cluster... 2023-04-12 02:54:19,917 INFO [PEWorker-1] procedure2.ProcedureExecutor(1357): LOCK_EVENT_WAIT rollback...pid=44, state=FAILED:SPLIT_TABLE_REGION_PREPARE, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=96cdae2f7638212730b694522b9a464a, daughterA=fdd18e9cf539c13d863267ce16031583, daughterB=578095173fbc35e181cce14464037c2e 2023-04-12 02:54:19,919 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=96cdae2f7638212730b694522b9a464a, ASSIGN 2023-04-12 02:54:19,921 DEBUG [Listener at localhost/46667] assignment.RegionStateStore(147): Load hbase:meta entry region=1f255e980a31bb266f2206a30aa2ae3f, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,44679,1681268031772, regionLocation=jenkins-hbase4.apache.org,44679,1681268031772, openSeqNum=2 2023-04-12 02:54:19,921 DEBUG [Listener at localhost/46667] assignment.RegionStateStore(147): Load hbase:meta entry region=96cdae2f7638212730b694522b9a464a, regionState=CLOSED, lastHost=jenkins-hbase4.apache.org,44679,1681268031772, regionLocation=jenkins-hbase4.apache.org,44679,1681268031772, openSeqNum=2 2023-04-12 02:54:19,921 INFO [Listener at localhost/46667] assignment.AssignmentManager(1529): Number of RegionServers=3 2023-04-12 02:54:19,921 INFO [Listener at localhost/46667] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1681268119921 2023-04-12 02:54:19,921 INFO [Listener at localhost/46667] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1681268179921 2023-04-12 02:54:19,921 INFO [Listener at localhost/46667] assignment.AssignmentManager(1536): Joined the cluster in 4 msec 2023-04-12 02:54:19,922 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-04-12 02:54:19,924 INFO [PEWorker-1] procedure2.ProcedureExecutor(1528): Rolled back pid=44, state=ROLLEDBACK, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=96cdae2f7638212730b694522b9a464a, daughterA=fdd18e9cf539c13d863267ce16031583, daughterB=578095173fbc35e181cce14464037c2e exec-time=5.5800 sec 2023-04-12 02:54:19,924 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=96cdae2f7638212730b694522b9a464a, ASSIGN 2023-04-12 02:54:19,925 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=96cdae2f7638212730b694522b9a464a, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,44679,1681268031772; forceNewPlan=false, retain=true 2023-04-12 02:54:20,075 INFO [jenkins-hbase4:40281] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-04-12 02:54:20,076 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=47 updating hbase:meta row=96cdae2f7638212730b694522b9a464a, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:20,076 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRollbackAndDoubleExecution,,1681268051553.96cdae2f7638212730b694522b9a464a.","families":{"info":[{"qualifier":"regioninfo","vlen":64,"tag":[],"timestamp":"1681268060076"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681268060076"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681268060076"}]},"ts":"1681268060076"} 2023-04-12 02:54:20,078 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=48, ppid=47, state=RUNNABLE; OpenRegionProcedure 96cdae2f7638212730b694522b9a464a, server=jenkins-hbase4.apache.org,44679,1681268031772}] 2023-04-12 02:54:20,078 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-04-12 02:54:20,078 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=47, state=WAITING:REGION_STATE_TRANSITION_CONFIRM_OPENED, locked=true; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=96cdae2f7638212730b694522b9a464a, ASSIGN 2023-04-12 02:54:20,078 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-04-12 02:54:20,078 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=47, state=WAITING:REGION_STATE_TRANSITION_CONFIRM_OPENED, locked=true; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=96cdae2f7638212730b694522b9a464a, ASSIGN at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-04-12 02:54:20,172 WARN [Listener at localhost/46667] procedure2.ProcedureTestingUtility(193): Set Kill before store update to: false 2023-04-12 02:54:20,172 WARN [Listener at localhost/46667] procedure.MasterProcedureTestingUtility(526): ProcedureExecutor not running, may have been stopped by pending procedure due to KillAndToggleBeforeStoreUpdate flag. 2023-04-12 02:54:20,172 INFO [Listener at localhost/46667] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-04-12 02:54:20,172 INFO [Listener at localhost/46667] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-04-12 02:54:20,172 INFO [Listener at localhost/46667] assignment.AssignmentManager(315): Stopping assignment manager 2023-04-12 02:54:20,172 ERROR [Listener at localhost/46667] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-04-12 02:54:20,172 INFO [Listener at localhost/46667] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-04-12 02:54:20,172 INFO [Listener at localhost/46667] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-04-12 02:54:20,172 INFO [Listener at localhost/46667] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-04-12 02:54:20,172 INFO [Listener at localhost/46667] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-04-12 02:54:20,173 INFO [Listener at localhost/46667] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-04-12 02:54:20,174 DEBUG [Listener at localhost/46667] store.ProcedureTree(114): Procedure Procedure(pid=47, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure) stack ids=[0] 2023-04-12 02:54:20,174 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-04-12 02:54:20,175 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-04-12 02:54:20,175 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-04-12 02:54:20,175 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-04-12 02:54:20,175 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-12 02:54:20,175 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=cc9012ec0b656baa24d04368fbe026c4, daughterA=e2a9b8fe15a781798c0b49e48ff5aad7, daughterB=6f55b5d6a460bd8c7569dee22f049104 2023-04-12 02:54:20,176 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-12 02:54:20,176 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-12 02:54:20,176 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-04-12 02:54:20,176 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=28, state=SUCCESS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=315e13e868a7f68a3ce37802b2f6312d, daughterA=4e1092f3c00484f8f09c730861225d43, daughterB=291734c00d212fd203b04cf8872d4649 2023-04-12 02:54:20,176 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=35, state=SUCCESS; DisableTableProcedure table=testSplitWithoutPONR 2023-04-12 02:54:20,176 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=40, state=SUCCESS; DeleteTableProcedure table=testSplitWithoutPONR 2023-04-12 02:54:20,177 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=41, state=SUCCESS; CreateTableProcedure table=testRollbackAndDoubleExecution 2023-04-12 02:54:20,177 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=44, state=ROLLEDBACK, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=96cdae2f7638212730b694522b9a464a, daughterA=fdd18e9cf539c13d863267ce16031583, daughterB=578095173fbc35e181cce14464037c2e 2023-04-12 02:54:20,177 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(468): Loading pid=47, state=RUNNABLE:REGION_STATE_TRANSITION_OPEN; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=96cdae2f7638212730b694522b9a464a, ASSIGN 2023-04-12 02:54:20,177 DEBUG [Listener at localhost/46667] procedure2.Procedure(966): pid=47, state=RUNNABLE:REGION_STATE_TRANSITION_OPEN; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=96cdae2f7638212730b694522b9a464a, ASSIGN held the lock before restarting, call acquireLock to restore it. 2023-04-12 02:54:20,177 INFO [Listener at localhost/46667] procedure.MasterProcedureScheduler(727): Took xlock for pid=47, state=RUNNABLE:REGION_STATE_TRANSITION_OPEN; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=96cdae2f7638212730b694522b9a464a, ASSIGN 2023-04-12 02:54:20,177 INFO [Listener at localhost/46667] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 4 msec 2023-04-12 02:54:20,178 DEBUG [Listener at localhost/46667] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-04-12 02:54:20,178 DEBUG [Listener at localhost/46667] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase4.apache.org,44679,1681268031772, table=hbase:meta, region=1588230740 2023-04-12 02:54:20,179 INFO [Listener at localhost/46667] assignment.AssignmentManager(305): Attach pid=47, state=RUNNABLE:REGION_STATE_TRANSITION_OPEN; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=96cdae2f7638212730b694522b9a464a, ASSIGN to state=OFFLINE, location=null, table=testRollbackAndDoubleExecution, region=96cdae2f7638212730b694522b9a464a to restore RIT 2023-04-12 02:54:20,179 INFO [Listener at localhost/46667] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1681268090179 2023-04-12 02:54:20,180 DEBUG [Listener at localhost/46667] assignment.AssignmentManager(1517): Joining cluster... 2023-04-12 02:54:20,180 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=47, state=RUNNABLE:REGION_STATE_TRANSITION_OPEN; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=96cdae2f7638212730b694522b9a464a, ASSIGN 2023-04-12 02:54:20,183 DEBUG [Listener at localhost/46667] assignment.RegionStateStore(147): Load hbase:meta entry region=1f255e980a31bb266f2206a30aa2ae3f, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,44679,1681268031772, regionLocation=jenkins-hbase4.apache.org,44679,1681268031772, openSeqNum=2 2023-04-12 02:54:20,183 DEBUG [Listener at localhost/46667] assignment.RegionStateStore(147): Load hbase:meta entry region=96cdae2f7638212730b694522b9a464a, regionState=OPENING, lastHost=jenkins-hbase4.apache.org,44679,1681268031772, regionLocation=jenkins-hbase4.apache.org,44679,1681268031772, openSeqNum=2 2023-04-12 02:54:20,183 INFO [Listener at localhost/46667] assignment.AssignmentManager(1529): Number of RegionServers=3 2023-04-12 02:54:20,183 INFO [Listener at localhost/46667] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1681268120183 2023-04-12 02:54:20,183 INFO [Listener at localhost/46667] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1681268180183 2023-04-12 02:54:20,183 INFO [Listener at localhost/46667] assignment.AssignmentManager(1536): Joined the cluster in 4 msec 2023-04-12 02:54:20,183 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=47 updating hbase:meta row=96cdae2f7638212730b694522b9a464a, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:20,183 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRollbackAndDoubleExecution,,1681268051553.96cdae2f7638212730b694522b9a464a.","families":{"info":[{"qualifier":"regioninfo","vlen":64,"tag":[],"timestamp":"1681268060183"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681268060183"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681268060183"}]},"ts":"1681268060183"} 2023-04-12 02:54:20,185 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=48, ppid=47, state=RUNNABLE; OpenRegionProcedure 96cdae2f7638212730b694522b9a464a, server=jenkins-hbase4.apache.org,44679,1681268031772}] 2023-04-12 02:54:20,340 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRollbackAndDoubleExecution,,1681268051553.96cdae2f7638212730b694522b9a464a. 2023-04-12 02:54:20,340 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 96cdae2f7638212730b694522b9a464a, NAME => 'testRollbackAndDoubleExecution,,1681268051553.96cdae2f7638212730b694522b9a464a.', STARTKEY => '', ENDKEY => ''} 2023-04-12 02:54:20,341 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(174): System coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro loaded, priority=536870911. 2023-04-12 02:54:20,341 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRollbackAndDoubleExecution 96cdae2f7638212730b694522b9a464a 2023-04-12 02:54:20,341 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated testRollbackAndDoubleExecution,,1681268051553.96cdae2f7638212730b694522b9a464a.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-12 02:54:20,341 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 96cdae2f7638212730b694522b9a464a 2023-04-12 02:54:20,341 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 96cdae2f7638212730b694522b9a464a 2023-04-12 02:54:20,342 INFO [StoreOpener-96cdae2f7638212730b694522b9a464a-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf1 of region 96cdae2f7638212730b694522b9a464a 2023-04-12 02:54:20,343 DEBUG [StoreOpener-96cdae2f7638212730b694522b9a464a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/cf1 2023-04-12 02:54:20,343 DEBUG [StoreOpener-96cdae2f7638212730b694522b9a464a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/cf1 2023-04-12 02:54:20,344 INFO [StoreOpener-96cdae2f7638212730b694522b9a464a-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 96cdae2f7638212730b694522b9a464a columnFamilyName cf1 2023-04-12 02:54:20,349 DEBUG [StoreOpener-96cdae2f7638212730b694522b9a464a-1] regionserver.HStore(539): loaded hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/cf1/0693b446d83544e4a9700be8c15de027 2023-04-12 02:54:20,352 DEBUG [StoreOpener-96cdae2f7638212730b694522b9a464a-1] regionserver.HStore(539): loaded hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/cf1/0864a86174d140c0a957973bc4f9f7dc 2023-04-12 02:54:20,355 DEBUG [StoreOpener-96cdae2f7638212730b694522b9a464a-1] regionserver.HStore(539): loaded hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/cf1/091b8b6c71524b5ea1657e72778f8478 2023-04-12 02:54:20,359 DEBUG [StoreOpener-96cdae2f7638212730b694522b9a464a-1] regionserver.HStore(539): loaded hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/cf1/177232c91ccf4f5d91bf18c2d4d5022f 2023-04-12 02:54:20,362 DEBUG [StoreOpener-96cdae2f7638212730b694522b9a464a-1] regionserver.HStore(539): loaded hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/cf1/6fbe235c9d994e648689a95a2833784a 2023-04-12 02:54:20,365 DEBUG [StoreOpener-96cdae2f7638212730b694522b9a464a-1] regionserver.HStore(539): loaded hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/cf1/a3b5b9d8fa954923ae97ea97984ade02 2023-04-12 02:54:20,368 DEBUG [StoreOpener-96cdae2f7638212730b694522b9a464a-1] regionserver.HStore(539): loaded hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/cf1/b5b939a0ed784022a8f999fa3edea7b9 2023-04-12 02:54:20,368 INFO [StoreOpener-96cdae2f7638212730b694522b9a464a-1] regionserver.HStore(310): Store=96cdae2f7638212730b694522b9a464a/cf1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-12 02:54:20,368 INFO [StoreOpener-96cdae2f7638212730b694522b9a464a-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf2 of region 96cdae2f7638212730b694522b9a464a 2023-04-12 02:54:20,369 DEBUG [StoreOpener-96cdae2f7638212730b694522b9a464a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/cf2 2023-04-12 02:54:20,369 DEBUG [StoreOpener-96cdae2f7638212730b694522b9a464a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/cf2 2023-04-12 02:54:20,370 INFO [StoreOpener-96cdae2f7638212730b694522b9a464a-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 96cdae2f7638212730b694522b9a464a columnFamilyName cf2 2023-04-12 02:54:20,376 DEBUG [StoreOpener-96cdae2f7638212730b694522b9a464a-1] regionserver.HStore(539): loaded hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/cf2/11c4addd69d049beb33f17d53e40ab23 2023-04-12 02:54:20,382 DEBUG [StoreOpener-96cdae2f7638212730b694522b9a464a-1] regionserver.HStore(539): loaded hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/cf2/28ceff5b76a942bd8ce4bf8f9ffcc8c9 2023-04-12 02:54:20,388 DEBUG [StoreOpener-96cdae2f7638212730b694522b9a464a-1] regionserver.HStore(539): loaded hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/cf2/68757554e4ad4422867a2933985cb078 2023-04-12 02:54:20,392 DEBUG [StoreOpener-96cdae2f7638212730b694522b9a464a-1] regionserver.HStore(539): loaded hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/cf2/6c612c6ffd3d42eab35c5ae09327ecf8 2023-04-12 02:54:20,395 DEBUG [StoreOpener-96cdae2f7638212730b694522b9a464a-1] regionserver.HStore(539): loaded hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/cf2/a0a6721d34874582969e0d913c5a3410 2023-04-12 02:54:20,399 DEBUG [StoreOpener-96cdae2f7638212730b694522b9a464a-1] regionserver.HStore(539): loaded hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/cf2/cd4efe4b5ad64256947de9915b5f6552 2023-04-12 02:54:20,402 DEBUG [StoreOpener-96cdae2f7638212730b694522b9a464a-1] regionserver.HStore(539): loaded hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/cf2/f66940ce4359474eac6c223cf1a8b486 2023-04-12 02:54:20,402 INFO [StoreOpener-96cdae2f7638212730b694522b9a464a-1] regionserver.HStore(310): Store=96cdae2f7638212730b694522b9a464a/cf2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-12 02:54:20,403 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a 2023-04-12 02:54:20,404 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a 2023-04-12 02:54:20,406 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(602): Cleaned up old failed split transaction detritus: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/.splits 2023-04-12 02:54:20,406 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testRollbackAndDoubleExecution descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-04-12 02:54:20,408 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 96cdae2f7638212730b694522b9a464a 2023-04-12 02:54:20,408 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 96cdae2f7638212730b694522b9a464a; next sequenceid=86; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=60797861, jitterRate=-0.09404127299785614}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-04-12 02:54:20,408 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 96cdae2f7638212730b694522b9a464a: 2023-04-12 02:54:20,409 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRollbackAndDoubleExecution,,1681268051553.96cdae2f7638212730b694522b9a464a., pid=48, masterSystemTime=1681268060337 2023-04-12 02:54:20,409 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: Opening Region; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 02:54:20,410 DEBUG [RS:1;jenkins-hbase4:44679-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-12 02:54:20,411 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: Opening Region; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 02:54:20,411 DEBUG [RS:1;jenkins-hbase4:44679-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-12 02:54:20,411 DEBUG [RS:1;jenkins-hbase4:44679-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 35728 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-04-12 02:54:20,411 DEBUG [RS:1;jenkins-hbase4:44679-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 35728 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-04-12 02:54:20,411 DEBUG [RS:1;jenkins-hbase4:44679-longCompactions-0] regionserver.HStore(1912): 96cdae2f7638212730b694522b9a464a/cf2 is initiating minor compaction (all files) 2023-04-12 02:54:20,412 DEBUG [RS:1;jenkins-hbase4:44679-shortCompactions-0] regionserver.HStore(1912): 96cdae2f7638212730b694522b9a464a/cf1 is initiating minor compaction (all files) 2023-04-12 02:54:20,412 INFO [RS:1;jenkins-hbase4:44679-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 96cdae2f7638212730b694522b9a464a/cf2 in testRollbackAndDoubleExecution,,1681268051553.96cdae2f7638212730b694522b9a464a. 2023-04-12 02:54:20,412 INFO [RS:1;jenkins-hbase4:44679-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 96cdae2f7638212730b694522b9a464a/cf1 in testRollbackAndDoubleExecution,,1681268051553.96cdae2f7638212730b694522b9a464a. 2023-04-12 02:54:20,412 INFO [RS:1;jenkins-hbase4:44679-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/cf2/28ceff5b76a942bd8ce4bf8f9ffcc8c9, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/cf2/f66940ce4359474eac6c223cf1a8b486, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/cf2/6c612c6ffd3d42eab35c5ae09327ecf8, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/cf2/11c4addd69d049beb33f17d53e40ab23, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/cf2/cd4efe4b5ad64256947de9915b5f6552, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/cf2/68757554e4ad4422867a2933985cb078, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/cf2/a0a6721d34874582969e0d913c5a3410] into tmpdir=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/.tmp, totalSize=34.9 K 2023-04-12 02:54:20,412 INFO [RS:1;jenkins-hbase4:44679-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/cf1/6fbe235c9d994e648689a95a2833784a, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/cf1/0693b446d83544e4a9700be8c15de027, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/cf1/091b8b6c71524b5ea1657e72778f8478, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/cf1/177232c91ccf4f5d91bf18c2d4d5022f, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/cf1/0864a86174d140c0a957973bc4f9f7dc, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/cf1/b5b939a0ed784022a8f999fa3edea7b9, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/cf1/a3b5b9d8fa954923ae97ea97984ade02] into tmpdir=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/.tmp, totalSize=34.9 K 2023-04-12 02:54:20,413 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRollbackAndDoubleExecution,,1681268051553.96cdae2f7638212730b694522b9a464a. 2023-04-12 02:54:20,413 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRollbackAndDoubleExecution,,1681268051553.96cdae2f7638212730b694522b9a464a. 2023-04-12 02:54:20,413 DEBUG [RS:1;jenkins-hbase4:44679-shortCompactions-0] compactions.Compactor(207): Compacting 6fbe235c9d994e648689a95a2833784a, keycount=2, bloomtype=ROW, size=4.8 K, encoding=NONE, compression=NONE, seqNum=6, earliestPutTs=1681268052057 2023-04-12 02:54:20,413 DEBUG [RS:1;jenkins-hbase4:44679-longCompactions-0] compactions.Compactor(207): Compacting 28ceff5b76a942bd8ce4bf8f9ffcc8c9, keycount=2, bloomtype=ROW, size=4.8 K, encoding=NONE, compression=NONE, seqNum=6, earliestPutTs=1681268052057 2023-04-12 02:54:20,414 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=47 updating hbase:meta row=96cdae2f7638212730b694522b9a464a, regionState=OPEN, openSeqNum=86, regionLocation=jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:20,414 DEBUG [RS:1;jenkins-hbase4:44679-shortCompactions-0] compactions.Compactor(207): Compacting 0693b446d83544e4a9700be8c15de027, keycount=10, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=19, earliestPutTs=1681268052290 2023-04-12 02:54:20,414 DEBUG [RS:1;jenkins-hbase4:44679-longCompactions-0] compactions.Compactor(207): Compacting f66940ce4359474eac6c223cf1a8b486, keycount=10, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=19, earliestPutTs=1681268052290 2023-04-12 02:54:20,414 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRollbackAndDoubleExecution,,1681268051553.96cdae2f7638212730b694522b9a464a.","families":{"info":[{"qualifier":"regioninfo","vlen":64,"tag":[],"timestamp":"1681268060414"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1681268060414"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681268060414"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681268060414"}]},"ts":"1681268060414"} 2023-04-12 02:54:20,414 DEBUG [RS:1;jenkins-hbase4:44679-shortCompactions-0] compactions.Compactor(207): Compacting 091b8b6c71524b5ea1657e72778f8478, keycount=10, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=32, earliestPutTs=1681268052570 2023-04-12 02:54:20,414 DEBUG [RS:1;jenkins-hbase4:44679-longCompactions-0] compactions.Compactor(207): Compacting 6c612c6ffd3d42eab35c5ae09327ecf8, keycount=10, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=32, earliestPutTs=1681268052570 2023-04-12 02:54:20,415 DEBUG [RS:1;jenkins-hbase4:44679-shortCompactions-0] compactions.Compactor(207): Compacting 177232c91ccf4f5d91bf18c2d4d5022f, keycount=10, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=45, earliestPutTs=1681268053229 2023-04-12 02:54:20,415 DEBUG [RS:1;jenkins-hbase4:44679-longCompactions-0] compactions.Compactor(207): Compacting 11c4addd69d049beb33f17d53e40ab23, keycount=10, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=45, earliestPutTs=1681268053229 2023-04-12 02:54:20,415 DEBUG [RS:1;jenkins-hbase4:44679-shortCompactions-0] compactions.Compactor(207): Compacting 0864a86174d140c0a957973bc4f9f7dc, keycount=10, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=58, earliestPutTs=1681268053468 2023-04-12 02:54:20,415 DEBUG [RS:1;jenkins-hbase4:44679-longCompactions-0] compactions.Compactor(207): Compacting cd4efe4b5ad64256947de9915b5f6552, keycount=10, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=58, earliestPutTs=1681268053468 2023-04-12 02:54:20,416 DEBUG [RS:1;jenkins-hbase4:44679-shortCompactions-0] compactions.Compactor(207): Compacting b5b939a0ed784022a8f999fa3edea7b9, keycount=10, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=71, earliestPutTs=1681268053690 2023-04-12 02:54:20,416 DEBUG [RS:1;jenkins-hbase4:44679-longCompactions-0] compactions.Compactor(207): Compacting 68757554e4ad4422867a2933985cb078, keycount=10, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=71, earliestPutTs=1681268053690 2023-04-12 02:54:20,416 DEBUG [RS:1;jenkins-hbase4:44679-shortCompactions-0] compactions.Compactor(207): Compacting a3b5b9d8fa954923ae97ea97984ade02, keycount=8, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=82, earliestPutTs=1681268054064 2023-04-12 02:54:20,416 DEBUG [RS:1;jenkins-hbase4:44679-longCompactions-0] compactions.Compactor(207): Compacting a0a6721d34874582969e0d913c5a3410, keycount=8, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=82, earliestPutTs=1681268054064 2023-04-12 02:54:20,420 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=48, resume processing ppid=47 2023-04-12 02:54:20,420 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=48, ppid=47, state=SUCCESS; OpenRegionProcedure 96cdae2f7638212730b694522b9a464a, server=jenkins-hbase4.apache.org,44679,1681268031772 in 233 msec 2023-04-12 02:54:20,429 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=47, state=SUCCESS; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=96cdae2f7638212730b694522b9a464a, ASSIGN in 1.5560 sec 2023-04-12 02:54:20,434 INFO [Listener at localhost/46667] procedure2.ProcedureTestingUtility(335): procId=44 exception: org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested 2023-04-12 02:54:20,443 DEBUG [Listener at localhost/46667] hbase.HBaseTestingUtility(3430): Waiting until all regions of table testRollbackAndDoubleExecution get assigned. Timeout = 1500ms 2023-04-12 02:54:20,445 INFO [Listener at localhost/46667] hbase.Waiter(180): Waiting up to [1,500] milli-secs(wait.for.ratio=[1]) 2023-04-12 02:54:20,458 INFO [Listener at localhost/46667] hbase.HBaseTestingUtility(3484): All regions for table testRollbackAndDoubleExecution assigned to meta. Checking AM states. 2023-04-12 02:54:20,459 INFO [Listener at localhost/46667] hbase.Waiter(180): Waiting up to [1,500] milli-secs(wait.for.ratio=[1]) 2023-04-12 02:54:20,459 INFO [Listener at localhost/46667] hbase.HBaseTestingUtility(3504): All regions for table testRollbackAndDoubleExecution assigned. 2023-04-12 02:54:20,462 INFO [RS:1;jenkins-hbase4:44679-longCompactions-0] throttle.PressureAwareThroughputController(145): 96cdae2f7638212730b694522b9a464a#cf2#compaction#50 average throughput is 1.72 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 02:54:20,474 INFO [RS:1;jenkins-hbase4:44679-shortCompactions-0] throttle.PressureAwareThroughputController(145): 96cdae2f7638212730b694522b9a464a#cf1#compaction#51 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 02:54:20,493 DEBUG [RS:1;jenkins-hbase4:44679-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/.tmp/cf2/6cb31a459aa5423cacfd30d9a4c29738 as hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/cf2/6cb31a459aa5423cacfd30d9a4c29738 2023-04-12 02:54:20,497 DEBUG [RS:1;jenkins-hbase4:44679-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/.tmp/cf1/e4fb9ecd8ade44a8b2f5a50e42e3d383 as hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/cf1/e4fb9ecd8ade44a8b2f5a50e42e3d383 2023-04-12 02:54:20,505 INFO [RS:1;jenkins-hbase4:44679-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 96cdae2f7638212730b694522b9a464a/cf2 of 96cdae2f7638212730b694522b9a464a into 6cb31a459aa5423cacfd30d9a4c29738(size=6.9 K), total size for store is 6.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 02:54:20,505 DEBUG [RS:1;jenkins-hbase4:44679-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 96cdae2f7638212730b694522b9a464a: 2023-04-12 02:54:20,505 INFO [RS:1;jenkins-hbase4:44679-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=testRollbackAndDoubleExecution,,1681268051553.96cdae2f7638212730b694522b9a464a., storeName=96cdae2f7638212730b694522b9a464a/cf2, priority=9, startTime=1681268060411; duration=0sec 2023-04-12 02:54:20,510 DEBUG [RS:1;jenkins-hbase4:44679-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 02:54:20,511 INFO [RS:1;jenkins-hbase4:44679-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 96cdae2f7638212730b694522b9a464a/cf1 of 96cdae2f7638212730b694522b9a464a into e4fb9ecd8ade44a8b2f5a50e42e3d383(size=6.9 K), total size for store is 6.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 02:54:20,511 DEBUG [RS:1;jenkins-hbase4:44679-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 96cdae2f7638212730b694522b9a464a: 2023-04-12 02:54:20,511 INFO [RS:1;jenkins-hbase4:44679-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=testRollbackAndDoubleExecution,,1681268051553.96cdae2f7638212730b694522b9a464a., storeName=96cdae2f7638212730b694522b9a464a/cf1, priority=9, startTime=1681268060409; duration=0sec 2023-04-12 02:54:20,511 DEBUG [RS:1;jenkins-hbase4:44679-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 02:54:20,523 WARN [Listener at localhost/46667] procedure2.ProcedureTestingUtility(193): Set Kill before store update to: false 2023-04-12 02:54:20,525 INFO [Listener at localhost/46667] client.HBaseAdmin$15(890): Started disable of testRollbackAndDoubleExecution 2023-04-12 02:54:20,526 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable testRollbackAndDoubleExecution 2023-04-12 02:54:20,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] procedure2.ProcedureExecutor(1029): Stored pid=49, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testRollbackAndDoubleExecution 2023-04-12 02:54:20,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] master.MasterRpcServices(1227): Checking to see if procedure is done pid=49 2023-04-12 02:54:20,531 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRollbackAndDoubleExecution","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681268060531"}]},"ts":"1681268060531"} 2023-04-12 02:54:20,532 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testRollbackAndDoubleExecution, state=DISABLING in hbase:meta 2023-04-12 02:54:20,534 INFO [PEWorker-1] procedure.DisableTableProcedure(293): Set testRollbackAndDoubleExecution to state=DISABLING 2023-04-12 02:54:20,535 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=50, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=96cdae2f7638212730b694522b9a464a, UNASSIGN}] 2023-04-12 02:54:20,536 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=50, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=96cdae2f7638212730b694522b9a464a, UNASSIGN 2023-04-12 02:54:20,537 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=50 updating hbase:meta row=96cdae2f7638212730b694522b9a464a, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:20,537 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRollbackAndDoubleExecution,,1681268051553.96cdae2f7638212730b694522b9a464a.","families":{"info":[{"qualifier":"regioninfo","vlen":64,"tag":[],"timestamp":"1681268060537"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681268060537"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681268060537"}]},"ts":"1681268060537"} 2023-04-12 02:54:20,538 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=51, ppid=50, state=RUNNABLE; CloseRegionProcedure 96cdae2f7638212730b694522b9a464a, server=jenkins-hbase4.apache.org,44679,1681268031772}] 2023-04-12 02:54:20,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] master.MasterRpcServices(1227): Checking to see if procedure is done pid=49 2023-04-12 02:54:20,690 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 96cdae2f7638212730b694522b9a464a 2023-04-12 02:54:20,690 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 96cdae2f7638212730b694522b9a464a, disabling compactions & flushes 2023-04-12 02:54:20,690 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region testRollbackAndDoubleExecution,,1681268051553.96cdae2f7638212730b694522b9a464a. 2023-04-12 02:54:20,690 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testRollbackAndDoubleExecution,,1681268051553.96cdae2f7638212730b694522b9a464a. 2023-04-12 02:54:20,690 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on testRollbackAndDoubleExecution,,1681268051553.96cdae2f7638212730b694522b9a464a. after waiting 0 ms 2023-04-12 02:54:20,690 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region testRollbackAndDoubleExecution,,1681268051553.96cdae2f7638212730b694522b9a464a. 2023-04-12 02:54:20,691 DEBUG [StoreCloser-testRollbackAndDoubleExecution,,1681268051553.96cdae2f7638212730b694522b9a464a.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/cf1/6fbe235c9d994e648689a95a2833784a, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/cf1/0693b446d83544e4a9700be8c15de027, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/cf1/091b8b6c71524b5ea1657e72778f8478, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/cf1/177232c91ccf4f5d91bf18c2d4d5022f, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/cf1/0864a86174d140c0a957973bc4f9f7dc, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/cf1/b5b939a0ed784022a8f999fa3edea7b9, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/cf1/a3b5b9d8fa954923ae97ea97984ade02] to archive 2023-04-12 02:54:20,692 DEBUG [StoreCloser-testRollbackAndDoubleExecution,,1681268051553.96cdae2f7638212730b694522b9a464a.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-12 02:54:20,694 DEBUG [StoreCloser-testRollbackAndDoubleExecution,,1681268051553.96cdae2f7638212730b694522b9a464a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/cf1/6fbe235c9d994e648689a95a2833784a to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/cf1/6fbe235c9d994e648689a95a2833784a 2023-04-12 02:54:20,695 DEBUG [StoreCloser-testRollbackAndDoubleExecution,,1681268051553.96cdae2f7638212730b694522b9a464a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/cf1/0693b446d83544e4a9700be8c15de027 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/cf1/0693b446d83544e4a9700be8c15de027 2023-04-12 02:54:20,696 DEBUG [StoreCloser-testRollbackAndDoubleExecution,,1681268051553.96cdae2f7638212730b694522b9a464a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/cf1/091b8b6c71524b5ea1657e72778f8478 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/cf1/091b8b6c71524b5ea1657e72778f8478 2023-04-12 02:54:20,697 DEBUG [StoreCloser-testRollbackAndDoubleExecution,,1681268051553.96cdae2f7638212730b694522b9a464a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/cf1/177232c91ccf4f5d91bf18c2d4d5022f to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/cf1/177232c91ccf4f5d91bf18c2d4d5022f 2023-04-12 02:54:20,698 DEBUG [StoreCloser-testRollbackAndDoubleExecution,,1681268051553.96cdae2f7638212730b694522b9a464a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/cf1/0864a86174d140c0a957973bc4f9f7dc to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/cf1/0864a86174d140c0a957973bc4f9f7dc 2023-04-12 02:54:20,699 DEBUG [StoreCloser-testRollbackAndDoubleExecution,,1681268051553.96cdae2f7638212730b694522b9a464a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/cf1/b5b939a0ed784022a8f999fa3edea7b9 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/cf1/b5b939a0ed784022a8f999fa3edea7b9 2023-04-12 02:54:20,701 DEBUG [StoreCloser-testRollbackAndDoubleExecution,,1681268051553.96cdae2f7638212730b694522b9a464a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/cf1/a3b5b9d8fa954923ae97ea97984ade02 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/cf1/a3b5b9d8fa954923ae97ea97984ade02 2023-04-12 02:54:20,705 DEBUG [StoreCloser-testRollbackAndDoubleExecution,,1681268051553.96cdae2f7638212730b694522b9a464a.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/cf2/28ceff5b76a942bd8ce4bf8f9ffcc8c9, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/cf2/f66940ce4359474eac6c223cf1a8b486, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/cf2/6c612c6ffd3d42eab35c5ae09327ecf8, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/cf2/11c4addd69d049beb33f17d53e40ab23, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/cf2/cd4efe4b5ad64256947de9915b5f6552, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/cf2/68757554e4ad4422867a2933985cb078, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/cf2/a0a6721d34874582969e0d913c5a3410] to archive 2023-04-12 02:54:20,706 DEBUG [StoreCloser-testRollbackAndDoubleExecution,,1681268051553.96cdae2f7638212730b694522b9a464a.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-12 02:54:20,707 DEBUG [StoreCloser-testRollbackAndDoubleExecution,,1681268051553.96cdae2f7638212730b694522b9a464a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/cf2/28ceff5b76a942bd8ce4bf8f9ffcc8c9 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/cf2/28ceff5b76a942bd8ce4bf8f9ffcc8c9 2023-04-12 02:54:20,708 DEBUG [StoreCloser-testRollbackAndDoubleExecution,,1681268051553.96cdae2f7638212730b694522b9a464a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/cf2/f66940ce4359474eac6c223cf1a8b486 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/cf2/f66940ce4359474eac6c223cf1a8b486 2023-04-12 02:54:20,710 DEBUG [StoreCloser-testRollbackAndDoubleExecution,,1681268051553.96cdae2f7638212730b694522b9a464a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/cf2/6c612c6ffd3d42eab35c5ae09327ecf8 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/cf2/6c612c6ffd3d42eab35c5ae09327ecf8 2023-04-12 02:54:20,711 DEBUG [StoreCloser-testRollbackAndDoubleExecution,,1681268051553.96cdae2f7638212730b694522b9a464a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/cf2/11c4addd69d049beb33f17d53e40ab23 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/cf2/11c4addd69d049beb33f17d53e40ab23 2023-04-12 02:54:20,711 DEBUG [StoreCloser-testRollbackAndDoubleExecution,,1681268051553.96cdae2f7638212730b694522b9a464a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/cf2/cd4efe4b5ad64256947de9915b5f6552 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/cf2/cd4efe4b5ad64256947de9915b5f6552 2023-04-12 02:54:20,712 DEBUG [StoreCloser-testRollbackAndDoubleExecution,,1681268051553.96cdae2f7638212730b694522b9a464a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/cf2/68757554e4ad4422867a2933985cb078 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/cf2/68757554e4ad4422867a2933985cb078 2023-04-12 02:54:20,713 DEBUG [StoreCloser-testRollbackAndDoubleExecution,,1681268051553.96cdae2f7638212730b694522b9a464a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/cf2/a0a6721d34874582969e0d913c5a3410 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/cf2/a0a6721d34874582969e0d913c5a3410 2023-04-12 02:54:20,717 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/recovered.edits/90.seqid, newMaxSeqId=90, maxSeqId=85 2023-04-12 02:54:20,717 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro 2023-04-12 02:54:20,717 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed testRollbackAndDoubleExecution,,1681268051553.96cdae2f7638212730b694522b9a464a. 2023-04-12 02:54:20,717 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 96cdae2f7638212730b694522b9a464a: 2023-04-12 02:54:20,719 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 96cdae2f7638212730b694522b9a464a 2023-04-12 02:54:20,719 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=50 updating hbase:meta row=96cdae2f7638212730b694522b9a464a, regionState=CLOSED 2023-04-12 02:54:20,719 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRollbackAndDoubleExecution,,1681268051553.96cdae2f7638212730b694522b9a464a.","families":{"info":[{"qualifier":"regioninfo","vlen":64,"tag":[],"timestamp":"1681268060719"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681268060719"}]},"ts":"1681268060719"} 2023-04-12 02:54:20,723 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=51, resume processing ppid=50 2023-04-12 02:54:20,723 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=51, ppid=50, state=SUCCESS; CloseRegionProcedure 96cdae2f7638212730b694522b9a464a, server=jenkins-hbase4.apache.org,44679,1681268031772 in 182 msec 2023-04-12 02:54:20,724 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=50, resume processing ppid=49 2023-04-12 02:54:20,724 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=50, ppid=49, state=SUCCESS; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=96cdae2f7638212730b694522b9a464a, UNASSIGN in 188 msec 2023-04-12 02:54:20,725 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRollbackAndDoubleExecution","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681268060725"}]},"ts":"1681268060725"} 2023-04-12 02:54:20,726 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testRollbackAndDoubleExecution, state=DISABLED in hbase:meta 2023-04-12 02:54:20,728 INFO [PEWorker-1] procedure.DisableTableProcedure(305): Set testRollbackAndDoubleExecution to state=DISABLED 2023-04-12 02:54:20,729 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=49, state=SUCCESS; DisableTableProcedure table=testRollbackAndDoubleExecution in 202 msec 2023-04-12 02:54:20,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] master.MasterRpcServices(1227): Checking to see if procedure is done pid=49 2023-04-12 02:54:20,833 INFO [Listener at localhost/46667] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testRollbackAndDoubleExecution, procId: 49 completed 2023-04-12 02:54:20,834 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete testRollbackAndDoubleExecution 2023-04-12 02:54:20,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] procedure2.ProcedureExecutor(1029): Stored pid=52, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=testRollbackAndDoubleExecution 2023-04-12 02:54:20,837 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=52, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=testRollbackAndDoubleExecution 2023-04-12 02:54:20,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] master.MasterRpcServices(1227): Checking to see if procedure is done pid=52 2023-04-12 02:54:20,838 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=52, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=testRollbackAndDoubleExecution 2023-04-12 02:54:20,841 DEBUG [HFileArchiver-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a 2023-04-12 02:54:20,842 DEBUG [HFileArchiver-5] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/cf1, FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/cf2, FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/recovered.edits] 2023-04-12 02:54:20,845 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/cf1/e4fb9ecd8ade44a8b2f5a50e42e3d383 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/cf1/e4fb9ecd8ade44a8b2f5a50e42e3d383 2023-04-12 02:54:20,847 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/cf2/6cb31a459aa5423cacfd30d9a4c29738 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/cf2/6cb31a459aa5423cacfd30d9a4c29738 2023-04-12 02:54:20,850 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/recovered.edits/90.seqid to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a/recovered.edits/90.seqid 2023-04-12 02:54:20,851 DEBUG [HFileArchiver-5] backup.HFileArchiver(596): Deleted hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testRollbackAndDoubleExecution/96cdae2f7638212730b694522b9a464a 2023-04-12 02:54:20,851 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived testRollbackAndDoubleExecution regions 2023-04-12 02:54:20,854 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=52, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=testRollbackAndDoubleExecution 2023-04-12 02:54:20,857 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of testRollbackAndDoubleExecution from hbase:meta 2023-04-12 02:54:20,858 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'testRollbackAndDoubleExecution' descriptor. 2023-04-12 02:54:20,859 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=52, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=testRollbackAndDoubleExecution 2023-04-12 02:54:20,860 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'testRollbackAndDoubleExecution' from region states. 2023-04-12 02:54:20,860 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRollbackAndDoubleExecution,,1681268051553.96cdae2f7638212730b694522b9a464a.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1681268060860"}]},"ts":"9223372036854775807"} 2023-04-12 02:54:20,861 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-12 02:54:20,861 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 96cdae2f7638212730b694522b9a464a, NAME => 'testRollbackAndDoubleExecution,,1681268051553.96cdae2f7638212730b694522b9a464a.', STARTKEY => '', ENDKEY => ''}] 2023-04-12 02:54:20,861 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'testRollbackAndDoubleExecution' as deleted. 2023-04-12 02:54:20,861 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRollbackAndDoubleExecution","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1681268060861"}]},"ts":"9223372036854775807"} 2023-04-12 02:54:20,863 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table testRollbackAndDoubleExecution state from META 2023-04-12 02:54:20,865 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=52, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=testRollbackAndDoubleExecution 2023-04-12 02:54:20,866 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=52, state=SUCCESS; DeleteTableProcedure table=testRollbackAndDoubleExecution in 31 msec 2023-04-12 02:54:20,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] master.MasterRpcServices(1227): Checking to see if procedure is done pid=52 2023-04-12 02:54:20,939 INFO [Listener at localhost/46667] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:testRollbackAndDoubleExecution, procId: 52 completed 2023-04-12 02:54:20,953 INFO [Listener at localhost/46667] hbase.ResourceChecker(175): after: master.assignment.TestSplitTableRegionProcedure#testRollbackAndDoubleExecution Thread=398 (was 402), OpenFileDescriptor=832 (was 826) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=462 (was 457) - SystemLoadAverage LEAK? -, ProcessCount=172 (was 172), AvailableMemoryMB=1132 (was 1224) 2023-04-12 02:54:20,967 INFO [Listener at localhost/46667] hbase.ResourceChecker(147): before: master.assignment.TestSplitTableRegionProcedure#testSplitTableRegionNoStoreFile Thread=398, OpenFileDescriptor=832, MaxFileDescriptor=60000, SystemLoadAverage=462, ProcessCount=172, AvailableMemoryMB=1131 2023-04-12 02:54:20,967 WARN [Listener at localhost/46667] procedure2.ProcedureTestingUtility(193): Set Kill before store update to: false 2023-04-12 02:54:20,970 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/balancer 2023-04-12 02:54:20,970 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] master.MasterRpcServices(492): Client=jenkins//172.31.14.131 set balanceSwitch=false 2023-04-12 02:54:20,971 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(1029): Stored pid=53, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=testSplitTableRegionNoStoreFile 2023-04-12 02:54:20,973 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=53, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=testSplitTableRegionNoStoreFile execute state=CREATE_TABLE_PRE_OPERATION 2023-04-12 02:54:20,973 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=53, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=testSplitTableRegionNoStoreFile execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-12 02:54:20,975 DEBUG [HFileArchiver-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitTableRegionNoStoreFile/947433d660677a65ad9876ce2025e3e0 2023-04-12 02:54:20,976 DEBUG [HFileArchiver-4] backup.HFileArchiver(153): Directory hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitTableRegionNoStoreFile/947433d660677a65ad9876ce2025e3e0 empty. 2023-04-12 02:54:20,976 DEBUG [HFileArchiver-4] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitTableRegionNoStoreFile/947433d660677a65ad9876ce2025e3e0 2023-04-12 02:54:20,976 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived testSplitTableRegionNoStoreFile regions 2023-04-12 02:54:20,985 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitTableRegionNoStoreFile/.tabledesc/.tableinfo.0000000001 2023-04-12 02:54:20,987 INFO [RegionOpenAndInit-testSplitTableRegionNoStoreFile-pool-0] regionserver.HRegion(7675): creating {ENCODED => 947433d660677a65ad9876ce2025e3e0, NAME => 'testSplitTableRegionNoStoreFile,,1681268060970.947433d660677a65ad9876ce2025e3e0.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='testSplitTableRegionNoStoreFile', {NAME => 'cf1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'cf2', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp 2023-04-12 02:54:21,003 DEBUG [RegionOpenAndInit-testSplitTableRegionNoStoreFile-pool-0] regionserver.HRegion(866): Instantiated testSplitTableRegionNoStoreFile,,1681268060970.947433d660677a65ad9876ce2025e3e0.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-12 02:54:21,003 DEBUG [RegionOpenAndInit-testSplitTableRegionNoStoreFile-pool-0] regionserver.HRegion(1604): Closing 947433d660677a65ad9876ce2025e3e0, disabling compactions & flushes 2023-04-12 02:54:21,003 INFO [RegionOpenAndInit-testSplitTableRegionNoStoreFile-pool-0] regionserver.HRegion(1626): Closing region testSplitTableRegionNoStoreFile,,1681268060970.947433d660677a65ad9876ce2025e3e0. 2023-04-12 02:54:21,003 DEBUG [RegionOpenAndInit-testSplitTableRegionNoStoreFile-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testSplitTableRegionNoStoreFile,,1681268060970.947433d660677a65ad9876ce2025e3e0. 2023-04-12 02:54:21,003 DEBUG [RegionOpenAndInit-testSplitTableRegionNoStoreFile-pool-0] regionserver.HRegion(1714): Acquired close lock on testSplitTableRegionNoStoreFile,,1681268060970.947433d660677a65ad9876ce2025e3e0. after waiting 0 ms 2023-04-12 02:54:21,003 DEBUG [RegionOpenAndInit-testSplitTableRegionNoStoreFile-pool-0] regionserver.HRegion(1724): Updates disabled for region testSplitTableRegionNoStoreFile,,1681268060970.947433d660677a65ad9876ce2025e3e0. 2023-04-12 02:54:21,003 INFO [RegionOpenAndInit-testSplitTableRegionNoStoreFile-pool-0] regionserver.HRegion(1838): Closed testSplitTableRegionNoStoreFile,,1681268060970.947433d660677a65ad9876ce2025e3e0. 2023-04-12 02:54:21,003 DEBUG [RegionOpenAndInit-testSplitTableRegionNoStoreFile-pool-0] regionserver.HRegion(1558): Region close journal for 947433d660677a65ad9876ce2025e3e0: 2023-04-12 02:54:21,006 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=53, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=testSplitTableRegionNoStoreFile execute state=CREATE_TABLE_ADD_TO_META 2023-04-12 02:54:21,007 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testSplitTableRegionNoStoreFile,,1681268060970.947433d660677a65ad9876ce2025e3e0.","families":{"info":[{"qualifier":"regioninfo","vlen":65,"tag":[],"timestamp":"1681268061007"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681268061007"}]},"ts":"1681268061007"} 2023-04-12 02:54:21,009 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-12 02:54:21,010 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=53, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=testSplitTableRegionNoStoreFile execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-12 02:54:21,010 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testSplitTableRegionNoStoreFile","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681268061010"}]},"ts":"1681268061010"} 2023-04-12 02:54:21,011 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testSplitTableRegionNoStoreFile, state=ENABLING in hbase:meta 2023-04-12 02:54:21,014 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-04-12 02:54:21,015 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-04-12 02:54:21,016 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-04-12 02:54:21,016 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-04-12 02:54:21,016 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-04-12 02:54:21,016 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=54, ppid=53, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSplitTableRegionNoStoreFile, region=947433d660677a65ad9876ce2025e3e0, ASSIGN}] 2023-04-12 02:54:21,018 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=54, ppid=53, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSplitTableRegionNoStoreFile, region=947433d660677a65ad9876ce2025e3e0, ASSIGN 2023-04-12 02:54:21,018 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=54, ppid=53, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testSplitTableRegionNoStoreFile, region=947433d660677a65ad9876ce2025e3e0, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,44679,1681268031772; forceNewPlan=false, retain=false 2023-04-12 02:54:21,169 INFO [jenkins-hbase4:40281] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-04-12 02:54:21,170 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=54 updating hbase:meta row=947433d660677a65ad9876ce2025e3e0, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:21,170 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitTableRegionNoStoreFile,,1681268060970.947433d660677a65ad9876ce2025e3e0.","families":{"info":[{"qualifier":"regioninfo","vlen":65,"tag":[],"timestamp":"1681268061170"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681268061170"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681268061170"}]},"ts":"1681268061170"} 2023-04-12 02:54:21,172 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=55, ppid=54, state=RUNNABLE; OpenRegionProcedure 947433d660677a65ad9876ce2025e3e0, server=jenkins-hbase4.apache.org,44679,1681268031772}] 2023-04-12 02:54:21,326 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testSplitTableRegionNoStoreFile,,1681268060970.947433d660677a65ad9876ce2025e3e0. 2023-04-12 02:54:21,327 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 947433d660677a65ad9876ce2025e3e0, NAME => 'testSplitTableRegionNoStoreFile,,1681268060970.947433d660677a65ad9876ce2025e3e0.', STARTKEY => '', ENDKEY => ''} 2023-04-12 02:54:21,327 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(174): System coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro loaded, priority=536870911. 2023-04-12 02:54:21,327 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testSplitTableRegionNoStoreFile 947433d660677a65ad9876ce2025e3e0 2023-04-12 02:54:21,327 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated testSplitTableRegionNoStoreFile,,1681268060970.947433d660677a65ad9876ce2025e3e0.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-12 02:54:21,327 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 947433d660677a65ad9876ce2025e3e0 2023-04-12 02:54:21,327 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 947433d660677a65ad9876ce2025e3e0 2023-04-12 02:54:21,328 INFO [StoreOpener-947433d660677a65ad9876ce2025e3e0-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf1 of region 947433d660677a65ad9876ce2025e3e0 2023-04-12 02:54:21,330 DEBUG [StoreOpener-947433d660677a65ad9876ce2025e3e0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionNoStoreFile/947433d660677a65ad9876ce2025e3e0/cf1 2023-04-12 02:54:21,330 DEBUG [StoreOpener-947433d660677a65ad9876ce2025e3e0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionNoStoreFile/947433d660677a65ad9876ce2025e3e0/cf1 2023-04-12 02:54:21,330 INFO [StoreOpener-947433d660677a65ad9876ce2025e3e0-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 947433d660677a65ad9876ce2025e3e0 columnFamilyName cf1 2023-04-12 02:54:21,331 INFO [StoreOpener-947433d660677a65ad9876ce2025e3e0-1] regionserver.HStore(310): Store=947433d660677a65ad9876ce2025e3e0/cf1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-12 02:54:21,331 INFO [StoreOpener-947433d660677a65ad9876ce2025e3e0-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf2 of region 947433d660677a65ad9876ce2025e3e0 2023-04-12 02:54:21,333 DEBUG [StoreOpener-947433d660677a65ad9876ce2025e3e0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionNoStoreFile/947433d660677a65ad9876ce2025e3e0/cf2 2023-04-12 02:54:21,333 DEBUG [StoreOpener-947433d660677a65ad9876ce2025e3e0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionNoStoreFile/947433d660677a65ad9876ce2025e3e0/cf2 2023-04-12 02:54:21,333 INFO [StoreOpener-947433d660677a65ad9876ce2025e3e0-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 947433d660677a65ad9876ce2025e3e0 columnFamilyName cf2 2023-04-12 02:54:21,334 INFO [StoreOpener-947433d660677a65ad9876ce2025e3e0-1] regionserver.HStore(310): Store=947433d660677a65ad9876ce2025e3e0/cf2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-12 02:54:21,334 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionNoStoreFile/947433d660677a65ad9876ce2025e3e0 2023-04-12 02:54:21,335 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionNoStoreFile/947433d660677a65ad9876ce2025e3e0 2023-04-12 02:54:21,336 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testSplitTableRegionNoStoreFile descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-04-12 02:54:21,338 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 947433d660677a65ad9876ce2025e3e0 2023-04-12 02:54:21,340 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionNoStoreFile/947433d660677a65ad9876ce2025e3e0/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-12 02:54:21,340 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 947433d660677a65ad9876ce2025e3e0; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=60375489, jitterRate=-0.10033510625362396}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-04-12 02:54:21,340 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 947433d660677a65ad9876ce2025e3e0: 2023-04-12 02:54:21,341 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testSplitTableRegionNoStoreFile,,1681268060970.947433d660677a65ad9876ce2025e3e0., pid=55, masterSystemTime=1681268061323 2023-04-12 02:54:21,342 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testSplitTableRegionNoStoreFile,,1681268060970.947433d660677a65ad9876ce2025e3e0. 2023-04-12 02:54:21,342 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testSplitTableRegionNoStoreFile,,1681268060970.947433d660677a65ad9876ce2025e3e0. 2023-04-12 02:54:21,343 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=54 updating hbase:meta row=947433d660677a65ad9876ce2025e3e0, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:21,343 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testSplitTableRegionNoStoreFile,,1681268060970.947433d660677a65ad9876ce2025e3e0.","families":{"info":[{"qualifier":"regioninfo","vlen":65,"tag":[],"timestamp":"1681268061343"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1681268061343"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681268061343"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681268061343"}]},"ts":"1681268061343"} 2023-04-12 02:54:21,345 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=55, resume processing ppid=54 2023-04-12 02:54:21,345 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=55, ppid=54, state=SUCCESS; OpenRegionProcedure 947433d660677a65ad9876ce2025e3e0, server=jenkins-hbase4.apache.org,44679,1681268031772 in 172 msec 2023-04-12 02:54:21,347 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=54, resume processing ppid=53 2023-04-12 02:54:21,347 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=54, ppid=53, state=SUCCESS; TransitRegionStateProcedure table=testSplitTableRegionNoStoreFile, region=947433d660677a65ad9876ce2025e3e0, ASSIGN in 330 msec 2023-04-12 02:54:21,348 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=53, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=testSplitTableRegionNoStoreFile execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-12 02:54:21,348 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testSplitTableRegionNoStoreFile","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681268061348"}]},"ts":"1681268061348"} 2023-04-12 02:54:21,349 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testSplitTableRegionNoStoreFile, state=ENABLED in hbase:meta 2023-04-12 02:54:21,351 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=53, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=testSplitTableRegionNoStoreFile execute state=CREATE_TABLE_POST_OPERATION 2023-04-12 02:54:21,352 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=53, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionNoStoreFile in 381 msec 2023-04-12 02:54:21,472 DEBUG [Listener at localhost/46667] assignment.SplitTableRegionProcedure(219): Splittable=true state=OPEN, location=jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:21,473 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(1029): Stored pid=56, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testSplitTableRegionNoStoreFile, parent=947433d660677a65ad9876ce2025e3e0, daughterA=b24cea6994eaa39f3128573197d02d9b, daughterB=b0e3d5fd7da33cb33084791956f0d7ca 2023-04-12 02:54:21,474 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=56, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testSplitTableRegionNoStoreFile, parent=947433d660677a65ad9876ce2025e3e0, daughterA=b24cea6994eaa39f3128573197d02d9b, daughterB=b0e3d5fd7da33cb33084791956f0d7ca 2023-04-12 02:54:21,474 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=56, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testSplitTableRegionNoStoreFile, parent=947433d660677a65ad9876ce2025e3e0, daughterA=b24cea6994eaa39f3128573197d02d9b, daughterB=b0e3d5fd7da33cb33084791956f0d7ca 2023-04-12 02:54:21,474 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=56, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testSplitTableRegionNoStoreFile, parent=947433d660677a65ad9876ce2025e3e0, daughterA=b24cea6994eaa39f3128573197d02d9b, daughterB=b0e3d5fd7da33cb33084791956f0d7ca 2023-04-12 02:54:21,476 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=57, ppid=56, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitTableRegionNoStoreFile, region=947433d660677a65ad9876ce2025e3e0, UNASSIGN}] 2023-04-12 02:54:21,477 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=57, ppid=56, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitTableRegionNoStoreFile, region=947433d660677a65ad9876ce2025e3e0, UNASSIGN 2023-04-12 02:54:21,477 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=57 updating hbase:meta row=947433d660677a65ad9876ce2025e3e0, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:21,478 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitTableRegionNoStoreFile,,1681268060970.947433d660677a65ad9876ce2025e3e0.","families":{"info":[{"qualifier":"regioninfo","vlen":65,"tag":[],"timestamp":"1681268061477"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681268061477"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681268061477"}]},"ts":"1681268061477"} 2023-04-12 02:54:21,479 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=58, ppid=57, state=RUNNABLE; CloseRegionProcedure 947433d660677a65ad9876ce2025e3e0, server=jenkins-hbase4.apache.org,44679,1681268031772}] 2023-04-12 02:54:21,631 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 947433d660677a65ad9876ce2025e3e0 2023-04-12 02:54:21,632 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 947433d660677a65ad9876ce2025e3e0, disabling compactions & flushes 2023-04-12 02:54:21,632 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region testSplitTableRegionNoStoreFile,,1681268060970.947433d660677a65ad9876ce2025e3e0. 2023-04-12 02:54:21,632 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testSplitTableRegionNoStoreFile,,1681268060970.947433d660677a65ad9876ce2025e3e0. 2023-04-12 02:54:21,632 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on testSplitTableRegionNoStoreFile,,1681268060970.947433d660677a65ad9876ce2025e3e0. after waiting 0 ms 2023-04-12 02:54:21,632 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region testSplitTableRegionNoStoreFile,,1681268060970.947433d660677a65ad9876ce2025e3e0. 2023-04-12 02:54:21,635 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionNoStoreFile/947433d660677a65ad9876ce2025e3e0/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-04-12 02:54:21,636 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro 2023-04-12 02:54:21,636 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed testSplitTableRegionNoStoreFile,,1681268060970.947433d660677a65ad9876ce2025e3e0. 2023-04-12 02:54:21,636 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 947433d660677a65ad9876ce2025e3e0: 2023-04-12 02:54:21,638 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 947433d660677a65ad9876ce2025e3e0 2023-04-12 02:54:21,638 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=57 updating hbase:meta row=947433d660677a65ad9876ce2025e3e0, regionState=CLOSED 2023-04-12 02:54:21,638 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testSplitTableRegionNoStoreFile,,1681268060970.947433d660677a65ad9876ce2025e3e0.","families":{"info":[{"qualifier":"regioninfo","vlen":65,"tag":[],"timestamp":"1681268061638"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681268061638"}]},"ts":"1681268061638"} 2023-04-12 02:54:21,641 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=58, resume processing ppid=57 2023-04-12 02:54:21,641 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=58, ppid=57, state=SUCCESS; CloseRegionProcedure 947433d660677a65ad9876ce2025e3e0, server=jenkins-hbase4.apache.org,44679,1681268031772 in 160 msec 2023-04-12 02:54:21,643 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=57, resume processing ppid=56 2023-04-12 02:54:21,643 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=57, ppid=56, state=SUCCESS; TransitRegionStateProcedure table=testSplitTableRegionNoStoreFile, region=947433d660677a65ad9876ce2025e3e0, UNASSIGN in 165 msec 2023-04-12 02:54:21,667 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionNoStoreFile/b24cea6994eaa39f3128573197d02d9b/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=-1 2023-04-12 02:54:21,669 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionNoStoreFile/b0e3d5fd7da33cb33084791956f0d7ca/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=-1 2023-04-12 02:54:21,671 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":3,"row":"testSplitTableRegionNoStoreFile,,1681268060970.947433d660677a65ad9876ce2025e3e0.","families":{"info":[{"qualifier":"regioninfo","vlen":65,"tag":[],"timestamp":"1681268061670"},{"qualifier":"splitA","vlen":67,"tag":[],"timestamp":"1681268061670"},{"qualifier":"splitB","vlen":67,"tag":[],"timestamp":"1681268061670"}]},"ts":"1681268061670"} 2023-04-12 02:54:21,671 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":3,"row":"testSplitTableRegionNoStoreFile,,1681268061472.b24cea6994eaa39f3128573197d02d9b.","families":{"info":[{"qualifier":"regioninfo","vlen":67,"tag":[],"timestamp":"1681268061670"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681268061670"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681268061670"}]},"ts":"1681268061670"} 2023-04-12 02:54:21,671 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":3,"row":"testSplitTableRegionNoStoreFile,41,1681268061472.b0e3d5fd7da33cb33084791956f0d7ca.","families":{"info":[{"qualifier":"regioninfo","vlen":67,"tag":[],"timestamp":"1681268061670"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681268061670"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681268061670"}]},"ts":"1681268061670"} 2023-04-12 02:54:21,674 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=59, ppid=56, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSplitTableRegionNoStoreFile, region=b24cea6994eaa39f3128573197d02d9b, ASSIGN}, {pid=60, ppid=56, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSplitTableRegionNoStoreFile, region=b0e3d5fd7da33cb33084791956f0d7ca, ASSIGN}] 2023-04-12 02:54:21,675 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=60, ppid=56, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSplitTableRegionNoStoreFile, region=b0e3d5fd7da33cb33084791956f0d7ca, ASSIGN 2023-04-12 02:54:21,676 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=60, ppid=56, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testSplitTableRegionNoStoreFile, region=b0e3d5fd7da33cb33084791956f0d7ca, ASSIGN; state=SPLITTING_NEW, location=jenkins-hbase4.apache.org,44679,1681268031772; forceNewPlan=false, retain=false 2023-04-12 02:54:21,676 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=59, ppid=56, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSplitTableRegionNoStoreFile, region=b24cea6994eaa39f3128573197d02d9b, ASSIGN 2023-04-12 02:54:21,677 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=59, ppid=56, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testSplitTableRegionNoStoreFile, region=b24cea6994eaa39f3128573197d02d9b, ASSIGN; state=SPLITTING_NEW, location=jenkins-hbase4.apache.org,44679,1681268031772; forceNewPlan=false, retain=false 2023-04-12 02:54:21,826 INFO [jenkins-hbase4:40281] balancer.BaseLoadBalancer(1545): Reassigned 2 regions. 2 retained the pre-restart assignment. 2023-04-12 02:54:21,828 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=60 updating hbase:meta row=b0e3d5fd7da33cb33084791956f0d7ca, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:21,828 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitTableRegionNoStoreFile,41,1681268061472.b0e3d5fd7da33cb33084791956f0d7ca.","families":{"info":[{"qualifier":"regioninfo","vlen":67,"tag":[],"timestamp":"1681268061828"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681268061828"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681268061828"}]},"ts":"1681268061828"} 2023-04-12 02:54:21,830 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=61, ppid=60, state=RUNNABLE; OpenRegionProcedure b0e3d5fd7da33cb33084791956f0d7ca, server=jenkins-hbase4.apache.org,44679,1681268031772}] 2023-04-12 02:54:21,832 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=b24cea6994eaa39f3128573197d02d9b, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:21,832 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitTableRegionNoStoreFile,,1681268061472.b24cea6994eaa39f3128573197d02d9b.","families":{"info":[{"qualifier":"regioninfo","vlen":67,"tag":[],"timestamp":"1681268061832"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681268061832"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681268061832"}]},"ts":"1681268061832"} 2023-04-12 02:54:21,833 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=62, ppid=59, state=RUNNABLE; OpenRegionProcedure b24cea6994eaa39f3128573197d02d9b, server=jenkins-hbase4.apache.org,44679,1681268031772}] 2023-04-12 02:54:21,984 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testSplitTableRegionNoStoreFile,,1681268061472.b24cea6994eaa39f3128573197d02d9b. 2023-04-12 02:54:21,984 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => b24cea6994eaa39f3128573197d02d9b, NAME => 'testSplitTableRegionNoStoreFile,,1681268061472.b24cea6994eaa39f3128573197d02d9b.', STARTKEY => '', ENDKEY => '41'} 2023-04-12 02:54:21,985 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(174): System coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro loaded, priority=536870911. 2023-04-12 02:54:21,985 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testSplitTableRegionNoStoreFile b24cea6994eaa39f3128573197d02d9b 2023-04-12 02:54:21,985 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated testSplitTableRegionNoStoreFile,,1681268061472.b24cea6994eaa39f3128573197d02d9b.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-12 02:54:21,985 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for b24cea6994eaa39f3128573197d02d9b 2023-04-12 02:54:21,985 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for b24cea6994eaa39f3128573197d02d9b 2023-04-12 02:54:21,986 INFO [StoreOpener-b24cea6994eaa39f3128573197d02d9b-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf1 of region b24cea6994eaa39f3128573197d02d9b 2023-04-12 02:54:21,987 DEBUG [StoreOpener-b24cea6994eaa39f3128573197d02d9b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionNoStoreFile/b24cea6994eaa39f3128573197d02d9b/cf1 2023-04-12 02:54:21,987 DEBUG [StoreOpener-b24cea6994eaa39f3128573197d02d9b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionNoStoreFile/b24cea6994eaa39f3128573197d02d9b/cf1 2023-04-12 02:54:21,988 INFO [StoreOpener-b24cea6994eaa39f3128573197d02d9b-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region b24cea6994eaa39f3128573197d02d9b columnFamilyName cf1 2023-04-12 02:54:21,988 INFO [StoreOpener-b24cea6994eaa39f3128573197d02d9b-1] regionserver.HStore(310): Store=b24cea6994eaa39f3128573197d02d9b/cf1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-12 02:54:21,988 INFO [StoreOpener-b24cea6994eaa39f3128573197d02d9b-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf2 of region b24cea6994eaa39f3128573197d02d9b 2023-04-12 02:54:21,990 DEBUG [StoreOpener-b24cea6994eaa39f3128573197d02d9b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionNoStoreFile/b24cea6994eaa39f3128573197d02d9b/cf2 2023-04-12 02:54:21,990 DEBUG [StoreOpener-b24cea6994eaa39f3128573197d02d9b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionNoStoreFile/b24cea6994eaa39f3128573197d02d9b/cf2 2023-04-12 02:54:21,990 INFO [StoreOpener-b24cea6994eaa39f3128573197d02d9b-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region b24cea6994eaa39f3128573197d02d9b columnFamilyName cf2 2023-04-12 02:54:21,990 INFO [StoreOpener-b24cea6994eaa39f3128573197d02d9b-1] regionserver.HStore(310): Store=b24cea6994eaa39f3128573197d02d9b/cf2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-12 02:54:21,991 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionNoStoreFile/b24cea6994eaa39f3128573197d02d9b 2023-04-12 02:54:21,992 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionNoStoreFile/b24cea6994eaa39f3128573197d02d9b 2023-04-12 02:54:21,993 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testSplitTableRegionNoStoreFile descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-04-12 02:54:21,994 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for b24cea6994eaa39f3128573197d02d9b 2023-04-12 02:54:21,995 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened b24cea6994eaa39f3128573197d02d9b; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=72109268, jitterRate=0.0745118260383606}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-04-12 02:54:21,995 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for b24cea6994eaa39f3128573197d02d9b: 2023-04-12 02:54:21,995 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testSplitTableRegionNoStoreFile,,1681268061472.b24cea6994eaa39f3128573197d02d9b., pid=62, masterSystemTime=1681268061981 2023-04-12 02:54:21,997 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testSplitTableRegionNoStoreFile,,1681268061472.b24cea6994eaa39f3128573197d02d9b. 2023-04-12 02:54:21,997 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testSplitTableRegionNoStoreFile,,1681268061472.b24cea6994eaa39f3128573197d02d9b. 2023-04-12 02:54:21,997 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testSplitTableRegionNoStoreFile,41,1681268061472.b0e3d5fd7da33cb33084791956f0d7ca. 2023-04-12 02:54:21,997 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => b0e3d5fd7da33cb33084791956f0d7ca, NAME => 'testSplitTableRegionNoStoreFile,41,1681268061472.b0e3d5fd7da33cb33084791956f0d7ca.', STARTKEY => '41', ENDKEY => ''} 2023-04-12 02:54:21,997 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=b24cea6994eaa39f3128573197d02d9b, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:21,997 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(174): System coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro loaded, priority=536870911. 2023-04-12 02:54:21,997 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testSplitTableRegionNoStoreFile,,1681268061472.b24cea6994eaa39f3128573197d02d9b.","families":{"info":[{"qualifier":"regioninfo","vlen":67,"tag":[],"timestamp":"1681268061997"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1681268061997"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681268061997"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681268061997"}]},"ts":"1681268061997"} 2023-04-12 02:54:21,997 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testSplitTableRegionNoStoreFile b0e3d5fd7da33cb33084791956f0d7ca 2023-04-12 02:54:21,997 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated testSplitTableRegionNoStoreFile,41,1681268061472.b0e3d5fd7da33cb33084791956f0d7ca.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-12 02:54:21,997 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for b0e3d5fd7da33cb33084791956f0d7ca 2023-04-12 02:54:21,997 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for b0e3d5fd7da33cb33084791956f0d7ca 2023-04-12 02:54:21,999 INFO [StoreOpener-b0e3d5fd7da33cb33084791956f0d7ca-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf1 of region b0e3d5fd7da33cb33084791956f0d7ca 2023-04-12 02:54:22,000 DEBUG [StoreOpener-b0e3d5fd7da33cb33084791956f0d7ca-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionNoStoreFile/b0e3d5fd7da33cb33084791956f0d7ca/cf1 2023-04-12 02:54:22,000 DEBUG [StoreOpener-b0e3d5fd7da33cb33084791956f0d7ca-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionNoStoreFile/b0e3d5fd7da33cb33084791956f0d7ca/cf1 2023-04-12 02:54:22,000 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=62, resume processing ppid=59 2023-04-12 02:54:22,000 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=62, ppid=59, state=SUCCESS; OpenRegionProcedure b24cea6994eaa39f3128573197d02d9b, server=jenkins-hbase4.apache.org,44679,1681268031772 in 166 msec 2023-04-12 02:54:22,000 INFO [StoreOpener-b0e3d5fd7da33cb33084791956f0d7ca-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region b0e3d5fd7da33cb33084791956f0d7ca columnFamilyName cf1 2023-04-12 02:54:22,001 INFO [StoreOpener-b0e3d5fd7da33cb33084791956f0d7ca-1] regionserver.HStore(310): Store=b0e3d5fd7da33cb33084791956f0d7ca/cf1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-12 02:54:22,001 INFO [StoreOpener-b0e3d5fd7da33cb33084791956f0d7ca-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf2 of region b0e3d5fd7da33cb33084791956f0d7ca 2023-04-12 02:54:22,001 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=59, ppid=56, state=SUCCESS; TransitRegionStateProcedure table=testSplitTableRegionNoStoreFile, region=b24cea6994eaa39f3128573197d02d9b, ASSIGN in 326 msec 2023-04-12 02:54:22,002 DEBUG [StoreOpener-b0e3d5fd7da33cb33084791956f0d7ca-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionNoStoreFile/b0e3d5fd7da33cb33084791956f0d7ca/cf2 2023-04-12 02:54:22,002 DEBUG [StoreOpener-b0e3d5fd7da33cb33084791956f0d7ca-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionNoStoreFile/b0e3d5fd7da33cb33084791956f0d7ca/cf2 2023-04-12 02:54:22,002 INFO [StoreOpener-b0e3d5fd7da33cb33084791956f0d7ca-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region b0e3d5fd7da33cb33084791956f0d7ca columnFamilyName cf2 2023-04-12 02:54:22,003 INFO [StoreOpener-b0e3d5fd7da33cb33084791956f0d7ca-1] regionserver.HStore(310): Store=b0e3d5fd7da33cb33084791956f0d7ca/cf2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-12 02:54:22,003 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionNoStoreFile/b0e3d5fd7da33cb33084791956f0d7ca 2023-04-12 02:54:22,004 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionNoStoreFile/b0e3d5fd7da33cb33084791956f0d7ca 2023-04-12 02:54:22,006 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testSplitTableRegionNoStoreFile descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-04-12 02:54:22,007 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for b0e3d5fd7da33cb33084791956f0d7ca 2023-04-12 02:54:22,009 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened b0e3d5fd7da33cb33084791956f0d7ca; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=63789036, jitterRate=-0.04946929216384888}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-04-12 02:54:22,009 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for b0e3d5fd7da33cb33084791956f0d7ca: 2023-04-12 02:54:22,009 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testSplitTableRegionNoStoreFile,41,1681268061472.b0e3d5fd7da33cb33084791956f0d7ca., pid=61, masterSystemTime=1681268061981 2023-04-12 02:54:22,010 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testSplitTableRegionNoStoreFile,41,1681268061472.b0e3d5fd7da33cb33084791956f0d7ca. 2023-04-12 02:54:22,011 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testSplitTableRegionNoStoreFile,41,1681268061472.b0e3d5fd7da33cb33084791956f0d7ca. 2023-04-12 02:54:22,011 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=60 updating hbase:meta row=b0e3d5fd7da33cb33084791956f0d7ca, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:22,011 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testSplitTableRegionNoStoreFile,41,1681268061472.b0e3d5fd7da33cb33084791956f0d7ca.","families":{"info":[{"qualifier":"regioninfo","vlen":67,"tag":[],"timestamp":"1681268062011"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1681268062011"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681268062011"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681268062011"}]},"ts":"1681268062011"} 2023-04-12 02:54:22,014 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=61, resume processing ppid=60 2023-04-12 02:54:22,014 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=61, ppid=60, state=SUCCESS; OpenRegionProcedure b0e3d5fd7da33cb33084791956f0d7ca, server=jenkins-hbase4.apache.org,44679,1681268031772 in 182 msec 2023-04-12 02:54:22,015 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=60, resume processing ppid=56 2023-04-12 02:54:22,015 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=60, ppid=56, state=SUCCESS; TransitRegionStateProcedure table=testSplitTableRegionNoStoreFile, region=b0e3d5fd7da33cb33084791956f0d7ca, ASSIGN in 340 msec 2023-04-12 02:54:22,016 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=56, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionNoStoreFile, parent=947433d660677a65ad9876ce2025e3e0, daughterA=b24cea6994eaa39f3128573197d02d9b, daughterB=b0e3d5fd7da33cb33084791956f0d7ca in 543 msec 2023-04-12 02:54:22,228 DEBUG [Listener at localhost/46667] client.ClientScanner(240): Advancing internal scanner to startKey at '41', inclusive 2023-04-12 02:54:22,230 WARN [Listener at localhost/46667] procedure2.ProcedureTestingUtility(193): Set Kill before store update to: false 2023-04-12 02:54:22,232 INFO [Listener at localhost/46667] client.HBaseAdmin$15(890): Started disable of testSplitTableRegionNoStoreFile 2023-04-12 02:54:22,232 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable testSplitTableRegionNoStoreFile 2023-04-12 02:54:22,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] procedure2.ProcedureExecutor(1029): Stored pid=63, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testSplitTableRegionNoStoreFile 2023-04-12 02:54:22,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] master.MasterRpcServices(1227): Checking to see if procedure is done pid=63 2023-04-12 02:54:22,236 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testSplitTableRegionNoStoreFile","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681268062236"}]},"ts":"1681268062236"} 2023-04-12 02:54:22,238 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testSplitTableRegionNoStoreFile, state=DISABLING in hbase:meta 2023-04-12 02:54:22,240 INFO [PEWorker-1] procedure.DisableTableProcedure(293): Set testSplitTableRegionNoStoreFile to state=DISABLING 2023-04-12 02:54:22,241 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=64, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitTableRegionNoStoreFile, region=b24cea6994eaa39f3128573197d02d9b, UNASSIGN}, {pid=65, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitTableRegionNoStoreFile, region=b0e3d5fd7da33cb33084791956f0d7ca, UNASSIGN}] 2023-04-12 02:54:22,242 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=65, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitTableRegionNoStoreFile, region=b0e3d5fd7da33cb33084791956f0d7ca, UNASSIGN 2023-04-12 02:54:22,242 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=65 updating hbase:meta row=b0e3d5fd7da33cb33084791956f0d7ca, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:22,242 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitTableRegionNoStoreFile,41,1681268061472.b0e3d5fd7da33cb33084791956f0d7ca.","families":{"info":[{"qualifier":"regioninfo","vlen":67,"tag":[],"timestamp":"1681268062242"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681268062242"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681268062242"}]},"ts":"1681268062242"} 2023-04-12 02:54:22,244 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=66, ppid=65, state=RUNNABLE; CloseRegionProcedure b0e3d5fd7da33cb33084791956f0d7ca, server=jenkins-hbase4.apache.org,44679,1681268031772}] 2023-04-12 02:54:22,252 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=64, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitTableRegionNoStoreFile, region=b24cea6994eaa39f3128573197d02d9b, UNASSIGN 2023-04-12 02:54:22,253 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=64 updating hbase:meta row=b24cea6994eaa39f3128573197d02d9b, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:22,253 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitTableRegionNoStoreFile,,1681268061472.b24cea6994eaa39f3128573197d02d9b.","families":{"info":[{"qualifier":"regioninfo","vlen":67,"tag":[],"timestamp":"1681268062253"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681268062253"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681268062253"}]},"ts":"1681268062253"} 2023-04-12 02:54:22,254 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=67, ppid=64, state=RUNNABLE; CloseRegionProcedure b24cea6994eaa39f3128573197d02d9b, server=jenkins-hbase4.apache.org,44679,1681268031772}] 2023-04-12 02:54:22,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] master.MasterRpcServices(1227): Checking to see if procedure is done pid=63 2023-04-12 02:54:22,396 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close b0e3d5fd7da33cb33084791956f0d7ca 2023-04-12 02:54:22,397 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing b0e3d5fd7da33cb33084791956f0d7ca, disabling compactions & flushes 2023-04-12 02:54:22,398 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region testSplitTableRegionNoStoreFile,41,1681268061472.b0e3d5fd7da33cb33084791956f0d7ca. 2023-04-12 02:54:22,398 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testSplitTableRegionNoStoreFile,41,1681268061472.b0e3d5fd7da33cb33084791956f0d7ca. 2023-04-12 02:54:22,398 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on testSplitTableRegionNoStoreFile,41,1681268061472.b0e3d5fd7da33cb33084791956f0d7ca. after waiting 0 ms 2023-04-12 02:54:22,398 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region testSplitTableRegionNoStoreFile,41,1681268061472.b0e3d5fd7da33cb33084791956f0d7ca. 2023-04-12 02:54:22,401 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionNoStoreFile/b0e3d5fd7da33cb33084791956f0d7ca/recovered.edits/7.seqid, newMaxSeqId=7, maxSeqId=4 2023-04-12 02:54:22,402 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro 2023-04-12 02:54:22,402 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed testSplitTableRegionNoStoreFile,41,1681268061472.b0e3d5fd7da33cb33084791956f0d7ca. 2023-04-12 02:54:22,402 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for b0e3d5fd7da33cb33084791956f0d7ca: 2023-04-12 02:54:22,403 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed b0e3d5fd7da33cb33084791956f0d7ca 2023-04-12 02:54:22,404 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close b24cea6994eaa39f3128573197d02d9b 2023-04-12 02:54:22,405 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing b24cea6994eaa39f3128573197d02d9b, disabling compactions & flushes 2023-04-12 02:54:22,405 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region testSplitTableRegionNoStoreFile,,1681268061472.b24cea6994eaa39f3128573197d02d9b. 2023-04-12 02:54:22,405 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testSplitTableRegionNoStoreFile,,1681268061472.b24cea6994eaa39f3128573197d02d9b. 2023-04-12 02:54:22,405 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on testSplitTableRegionNoStoreFile,,1681268061472.b24cea6994eaa39f3128573197d02d9b. after waiting 0 ms 2023-04-12 02:54:22,405 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region testSplitTableRegionNoStoreFile,,1681268061472.b24cea6994eaa39f3128573197d02d9b. 2023-04-12 02:54:22,405 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=65 updating hbase:meta row=b0e3d5fd7da33cb33084791956f0d7ca, regionState=CLOSED 2023-04-12 02:54:22,405 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testSplitTableRegionNoStoreFile,41,1681268061472.b0e3d5fd7da33cb33084791956f0d7ca.","families":{"info":[{"qualifier":"regioninfo","vlen":67,"tag":[],"timestamp":"1681268062405"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681268062405"}]},"ts":"1681268062405"} 2023-04-12 02:54:22,409 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionNoStoreFile/b24cea6994eaa39f3128573197d02d9b/recovered.edits/7.seqid, newMaxSeqId=7, maxSeqId=4 2023-04-12 02:54:22,409 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=66, resume processing ppid=65 2023-04-12 02:54:22,409 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=66, ppid=65, state=SUCCESS; CloseRegionProcedure b0e3d5fd7da33cb33084791956f0d7ca, server=jenkins-hbase4.apache.org,44679,1681268031772 in 162 msec 2023-04-12 02:54:22,409 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro 2023-04-12 02:54:22,409 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed testSplitTableRegionNoStoreFile,,1681268061472.b24cea6994eaa39f3128573197d02d9b. 2023-04-12 02:54:22,409 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for b24cea6994eaa39f3128573197d02d9b: 2023-04-12 02:54:22,410 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=65, ppid=63, state=SUCCESS; TransitRegionStateProcedure table=testSplitTableRegionNoStoreFile, region=b0e3d5fd7da33cb33084791956f0d7ca, UNASSIGN in 168 msec 2023-04-12 02:54:22,411 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed b24cea6994eaa39f3128573197d02d9b 2023-04-12 02:54:22,411 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=64 updating hbase:meta row=b24cea6994eaa39f3128573197d02d9b, regionState=CLOSED 2023-04-12 02:54:22,412 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testSplitTableRegionNoStoreFile,,1681268061472.b24cea6994eaa39f3128573197d02d9b.","families":{"info":[{"qualifier":"regioninfo","vlen":67,"tag":[],"timestamp":"1681268062411"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681268062411"}]},"ts":"1681268062411"} 2023-04-12 02:54:22,415 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=67, resume processing ppid=64 2023-04-12 02:54:22,415 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=67, ppid=64, state=SUCCESS; CloseRegionProcedure b24cea6994eaa39f3128573197d02d9b, server=jenkins-hbase4.apache.org,44679,1681268031772 in 159 msec 2023-04-12 02:54:22,417 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=64, resume processing ppid=63 2023-04-12 02:54:22,417 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=64, ppid=63, state=SUCCESS; TransitRegionStateProcedure table=testSplitTableRegionNoStoreFile, region=b24cea6994eaa39f3128573197d02d9b, UNASSIGN in 174 msec 2023-04-12 02:54:22,418 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testSplitTableRegionNoStoreFile","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681268062418"}]},"ts":"1681268062418"} 2023-04-12 02:54:22,419 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testSplitTableRegionNoStoreFile, state=DISABLED in hbase:meta 2023-04-12 02:54:22,421 INFO [PEWorker-1] procedure.DisableTableProcedure(305): Set testSplitTableRegionNoStoreFile to state=DISABLED 2023-04-12 02:54:22,423 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=63, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionNoStoreFile in 190 msec 2023-04-12 02:54:22,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] master.MasterRpcServices(1227): Checking to see if procedure is done pid=63 2023-04-12 02:54:22,539 INFO [Listener at localhost/46667] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testSplitTableRegionNoStoreFile, procId: 63 completed 2023-04-12 02:54:22,539 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete testSplitTableRegionNoStoreFile 2023-04-12 02:54:22,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] procedure2.ProcedureExecutor(1029): Stored pid=68, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=testSplitTableRegionNoStoreFile 2023-04-12 02:54:22,542 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=68, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=testSplitTableRegionNoStoreFile 2023-04-12 02:54:22,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] master.MasterRpcServices(1227): Checking to see if procedure is done pid=68 2023-04-12 02:54:22,543 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=68, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=testSplitTableRegionNoStoreFile 2023-04-12 02:54:22,549 DEBUG [HFileArchiver-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitTableRegionNoStoreFile/947433d660677a65ad9876ce2025e3e0 2023-04-12 02:54:22,549 DEBUG [HFileArchiver-6] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitTableRegionNoStoreFile/b24cea6994eaa39f3128573197d02d9b 2023-04-12 02:54:22,549 DEBUG [HFileArchiver-7] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitTableRegionNoStoreFile/b0e3d5fd7da33cb33084791956f0d7ca 2023-04-12 02:54:22,551 DEBUG [HFileArchiver-6] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitTableRegionNoStoreFile/b24cea6994eaa39f3128573197d02d9b/cf1, FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitTableRegionNoStoreFile/b24cea6994eaa39f3128573197d02d9b/cf2, FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitTableRegionNoStoreFile/b24cea6994eaa39f3128573197d02d9b/recovered.edits] 2023-04-12 02:54:22,551 DEBUG [HFileArchiver-7] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitTableRegionNoStoreFile/b0e3d5fd7da33cb33084791956f0d7ca/cf1, FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitTableRegionNoStoreFile/b0e3d5fd7da33cb33084791956f0d7ca/cf2, FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitTableRegionNoStoreFile/b0e3d5fd7da33cb33084791956f0d7ca/recovered.edits] 2023-04-12 02:54:22,551 DEBUG [HFileArchiver-3] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitTableRegionNoStoreFile/947433d660677a65ad9876ce2025e3e0/cf1, FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitTableRegionNoStoreFile/947433d660677a65ad9876ce2025e3e0/cf2, FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitTableRegionNoStoreFile/947433d660677a65ad9876ce2025e3e0/recovered.edits] 2023-04-12 02:54:22,558 DEBUG [HFileArchiver-6] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitTableRegionNoStoreFile/b24cea6994eaa39f3128573197d02d9b/recovered.edits/7.seqid to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegionNoStoreFile/b24cea6994eaa39f3128573197d02d9b/recovered.edits/7.seqid 2023-04-12 02:54:22,558 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitTableRegionNoStoreFile/b0e3d5fd7da33cb33084791956f0d7ca/recovered.edits/7.seqid to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegionNoStoreFile/b0e3d5fd7da33cb33084791956f0d7ca/recovered.edits/7.seqid 2023-04-12 02:54:22,559 DEBUG [HFileArchiver-6] backup.HFileArchiver(596): Deleted hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitTableRegionNoStoreFile/b24cea6994eaa39f3128573197d02d9b 2023-04-12 02:54:22,559 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitTableRegionNoStoreFile/947433d660677a65ad9876ce2025e3e0/recovered.edits/4.seqid to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegionNoStoreFile/947433d660677a65ad9876ce2025e3e0/recovered.edits/4.seqid 2023-04-12 02:54:22,559 DEBUG [HFileArchiver-7] backup.HFileArchiver(596): Deleted hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitTableRegionNoStoreFile/b0e3d5fd7da33cb33084791956f0d7ca 2023-04-12 02:54:22,559 DEBUG [HFileArchiver-3] backup.HFileArchiver(596): Deleted hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitTableRegionNoStoreFile/947433d660677a65ad9876ce2025e3e0 2023-04-12 02:54:22,559 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived testSplitTableRegionNoStoreFile regions 2023-04-12 02:54:22,562 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=68, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=testSplitTableRegionNoStoreFile 2023-04-12 02:54:22,563 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 3 rows of testSplitTableRegionNoStoreFile from hbase:meta 2023-04-12 02:54:22,565 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'testSplitTableRegionNoStoreFile' descriptor. 2023-04-12 02:54:22,567 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=68, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=testSplitTableRegionNoStoreFile 2023-04-12 02:54:22,567 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'testSplitTableRegionNoStoreFile' from region states. 2023-04-12 02:54:22,567 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testSplitTableRegionNoStoreFile,,1681268060970.947433d660677a65ad9876ce2025e3e0.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1681268062567"}]},"ts":"9223372036854775807"} 2023-04-12 02:54:22,567 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testSplitTableRegionNoStoreFile,,1681268061472.b24cea6994eaa39f3128573197d02d9b.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1681268062567"}]},"ts":"9223372036854775807"} 2023-04-12 02:54:22,567 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testSplitTableRegionNoStoreFile,41,1681268061472.b0e3d5fd7da33cb33084791956f0d7ca.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1681268062567"}]},"ts":"9223372036854775807"} 2023-04-12 02:54:22,569 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 3 regions from META 2023-04-12 02:54:22,569 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 947433d660677a65ad9876ce2025e3e0, NAME => 'testSplitTableRegionNoStoreFile,,1681268060970.947433d660677a65ad9876ce2025e3e0.', STARTKEY => '', ENDKEY => ''}, {ENCODED => b24cea6994eaa39f3128573197d02d9b, NAME => 'testSplitTableRegionNoStoreFile,,1681268061472.b24cea6994eaa39f3128573197d02d9b.', STARTKEY => '', ENDKEY => '41'}, {ENCODED => b0e3d5fd7da33cb33084791956f0d7ca, NAME => 'testSplitTableRegionNoStoreFile,41,1681268061472.b0e3d5fd7da33cb33084791956f0d7ca.', STARTKEY => '41', ENDKEY => ''}] 2023-04-12 02:54:22,569 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'testSplitTableRegionNoStoreFile' as deleted. 2023-04-12 02:54:22,569 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testSplitTableRegionNoStoreFile","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1681268062569"}]},"ts":"9223372036854775807"} 2023-04-12 02:54:22,570 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table testSplitTableRegionNoStoreFile state from META 2023-04-12 02:54:22,572 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=68, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=testSplitTableRegionNoStoreFile 2023-04-12 02:54:22,573 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=68, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionNoStoreFile in 33 msec 2023-04-12 02:54:22,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] master.MasterRpcServices(1227): Checking to see if procedure is done pid=68 2023-04-12 02:54:22,644 INFO [Listener at localhost/46667] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:testSplitTableRegionNoStoreFile, procId: 68 completed 2023-04-12 02:54:22,659 INFO [Listener at localhost/46667] hbase.ResourceChecker(175): after: master.assignment.TestSplitTableRegionProcedure#testSplitTableRegionNoStoreFile Thread=402 (was 398) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1753845133_15 at /127.0.0.1:43146 [Waiting for operation #3] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1753845133_15 at /127.0.0.1:56960 [Waiting for operation #10] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x2d8012cd-metaLookup-shared--pool-8 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x2d8012cd-shared-pool-11 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x2d8012cd-shared-pool-12 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x2d8012cd-metaLookup-shared--pool-7 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) - Thread LEAK? -, OpenFileDescriptor=832 (was 832), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=425 (was 462), ProcessCount=172 (was 172), AvailableMemoryMB=1130 (was 1131) 2023-04-12 02:54:22,674 INFO [Listener at localhost/46667] hbase.ResourceChecker(147): before: master.assignment.TestSplitTableRegionProcedure#testSplitTableRegionDeletedRowsDaughter Thread=402, OpenFileDescriptor=832, MaxFileDescriptor=60000, SystemLoadAverage=425, ProcessCount=172, AvailableMemoryMB=1130 2023-04-12 02:54:22,674 WARN [Listener at localhost/46667] procedure2.ProcedureTestingUtility(193): Set Kill before store update to: false 2023-04-12 02:54:22,676 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/balancer 2023-04-12 02:54:22,676 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] master.MasterRpcServices(492): Client=jenkins//172.31.14.131 set balanceSwitch=false 2023-04-12 02:54:22,678 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(1029): Stored pid=69, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:22,680 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=69, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=testSplitTableRegionDeletedRowsDaughter execute state=CREATE_TABLE_PRE_OPERATION 2023-04-12 02:54:22,680 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=69, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=testSplitTableRegionDeletedRowsDaughter execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-12 02:54:22,682 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8 2023-04-12 02:54:22,682 DEBUG [HFileArchiver-1] backup.HFileArchiver(153): Directory hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8 empty. 2023-04-12 02:54:22,683 DEBUG [HFileArchiver-1] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8 2023-04-12 02:54:22,683 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived testSplitTableRegionDeletedRowsDaughter regions 2023-04-12 02:54:22,693 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/.tabledesc/.tableinfo.0000000001 2023-04-12 02:54:22,694 INFO [RegionOpenAndInit-testSplitTableRegionDeletedRowsDaughter-pool-0] regionserver.HRegion(7675): creating {ENCODED => 1c6b8f18205322cbb248ea4fb10155d8, NAME => 'testSplitTableRegionDeletedRowsDaughter,,1681268062677.1c6b8f18205322cbb248ea4fb10155d8.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='testSplitTableRegionDeletedRowsDaughter', {NAME => 'cf1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'cf2', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp 2023-04-12 02:54:22,706 DEBUG [RegionOpenAndInit-testSplitTableRegionDeletedRowsDaughter-pool-0] regionserver.HRegion(866): Instantiated testSplitTableRegionDeletedRowsDaughter,,1681268062677.1c6b8f18205322cbb248ea4fb10155d8.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-12 02:54:22,706 DEBUG [RegionOpenAndInit-testSplitTableRegionDeletedRowsDaughter-pool-0] regionserver.HRegion(1604): Closing 1c6b8f18205322cbb248ea4fb10155d8, disabling compactions & flushes 2023-04-12 02:54:22,706 INFO [RegionOpenAndInit-testSplitTableRegionDeletedRowsDaughter-pool-0] regionserver.HRegion(1626): Closing region testSplitTableRegionDeletedRowsDaughter,,1681268062677.1c6b8f18205322cbb248ea4fb10155d8. 2023-04-12 02:54:22,706 DEBUG [RegionOpenAndInit-testSplitTableRegionDeletedRowsDaughter-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testSplitTableRegionDeletedRowsDaughter,,1681268062677.1c6b8f18205322cbb248ea4fb10155d8. 2023-04-12 02:54:22,706 DEBUG [RegionOpenAndInit-testSplitTableRegionDeletedRowsDaughter-pool-0] regionserver.HRegion(1714): Acquired close lock on testSplitTableRegionDeletedRowsDaughter,,1681268062677.1c6b8f18205322cbb248ea4fb10155d8. after waiting 0 ms 2023-04-12 02:54:22,706 DEBUG [RegionOpenAndInit-testSplitTableRegionDeletedRowsDaughter-pool-0] regionserver.HRegion(1724): Updates disabled for region testSplitTableRegionDeletedRowsDaughter,,1681268062677.1c6b8f18205322cbb248ea4fb10155d8. 2023-04-12 02:54:22,706 INFO [RegionOpenAndInit-testSplitTableRegionDeletedRowsDaughter-pool-0] regionserver.HRegion(1838): Closed testSplitTableRegionDeletedRowsDaughter,,1681268062677.1c6b8f18205322cbb248ea4fb10155d8. 2023-04-12 02:54:22,706 DEBUG [RegionOpenAndInit-testSplitTableRegionDeletedRowsDaughter-pool-0] regionserver.HRegion(1558): Region close journal for 1c6b8f18205322cbb248ea4fb10155d8: 2023-04-12 02:54:22,708 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=69, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=testSplitTableRegionDeletedRowsDaughter execute state=CREATE_TABLE_ADD_TO_META 2023-04-12 02:54:22,709 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testSplitTableRegionDeletedRowsDaughter,,1681268062677.1c6b8f18205322cbb248ea4fb10155d8.","families":{"info":[{"qualifier":"regioninfo","vlen":73,"tag":[],"timestamp":"1681268062709"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681268062709"}]},"ts":"1681268062709"} 2023-04-12 02:54:22,710 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-12 02:54:22,711 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=69, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=testSplitTableRegionDeletedRowsDaughter execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-12 02:54:22,711 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testSplitTableRegionDeletedRowsDaughter","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681268062711"}]},"ts":"1681268062711"} 2023-04-12 02:54:22,712 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testSplitTableRegionDeletedRowsDaughter, state=ENABLING in hbase:meta 2023-04-12 02:54:22,716 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-04-12 02:54:22,717 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-04-12 02:54:22,717 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-04-12 02:54:22,717 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-04-12 02:54:22,717 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-04-12 02:54:22,717 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=70, ppid=69, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSplitTableRegionDeletedRowsDaughter, region=1c6b8f18205322cbb248ea4fb10155d8, ASSIGN}] 2023-04-12 02:54:22,719 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=70, ppid=69, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSplitTableRegionDeletedRowsDaughter, region=1c6b8f18205322cbb248ea4fb10155d8, ASSIGN 2023-04-12 02:54:22,720 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=70, ppid=69, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testSplitTableRegionDeletedRowsDaughter, region=1c6b8f18205322cbb248ea4fb10155d8, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,34579,1681268031719; forceNewPlan=false, retain=false 2023-04-12 02:54:22,870 INFO [jenkins-hbase4:40281] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-04-12 02:54:22,871 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=70 updating hbase:meta row=1c6b8f18205322cbb248ea4fb10155d8, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:22,871 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitTableRegionDeletedRowsDaughter,,1681268062677.1c6b8f18205322cbb248ea4fb10155d8.","families":{"info":[{"qualifier":"regioninfo","vlen":73,"tag":[],"timestamp":"1681268062871"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681268062871"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681268062871"}]},"ts":"1681268062871"} 2023-04-12 02:54:22,873 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=71, ppid=70, state=RUNNABLE; OpenRegionProcedure 1c6b8f18205322cbb248ea4fb10155d8, server=jenkins-hbase4.apache.org,34579,1681268031719}] 2023-04-12 02:54:23,030 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testSplitTableRegionDeletedRowsDaughter,,1681268062677.1c6b8f18205322cbb248ea4fb10155d8. 2023-04-12 02:54:23,031 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1c6b8f18205322cbb248ea4fb10155d8, NAME => 'testSplitTableRegionDeletedRowsDaughter,,1681268062677.1c6b8f18205322cbb248ea4fb10155d8.', STARTKEY => '', ENDKEY => ''} 2023-04-12 02:54:23,031 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(174): System coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro loaded, priority=536870911. 2023-04-12 02:54:23,031 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testSplitTableRegionDeletedRowsDaughter 1c6b8f18205322cbb248ea4fb10155d8 2023-04-12 02:54:23,031 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated testSplitTableRegionDeletedRowsDaughter,,1681268062677.1c6b8f18205322cbb248ea4fb10155d8.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-12 02:54:23,031 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 1c6b8f18205322cbb248ea4fb10155d8 2023-04-12 02:54:23,031 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 1c6b8f18205322cbb248ea4fb10155d8 2023-04-12 02:54:23,032 INFO [StoreOpener-1c6b8f18205322cbb248ea4fb10155d8-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf1 of region 1c6b8f18205322cbb248ea4fb10155d8 2023-04-12 02:54:23,034 DEBUG [StoreOpener-1c6b8f18205322cbb248ea4fb10155d8-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/cf1 2023-04-12 02:54:23,034 DEBUG [StoreOpener-1c6b8f18205322cbb248ea4fb10155d8-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/cf1 2023-04-12 02:54:23,034 INFO [StoreOpener-1c6b8f18205322cbb248ea4fb10155d8-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1c6b8f18205322cbb248ea4fb10155d8 columnFamilyName cf1 2023-04-12 02:54:23,035 INFO [StoreOpener-1c6b8f18205322cbb248ea4fb10155d8-1] regionserver.HStore(310): Store=1c6b8f18205322cbb248ea4fb10155d8/cf1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-12 02:54:23,035 INFO [StoreOpener-1c6b8f18205322cbb248ea4fb10155d8-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf2 of region 1c6b8f18205322cbb248ea4fb10155d8 2023-04-12 02:54:23,037 DEBUG [StoreOpener-1c6b8f18205322cbb248ea4fb10155d8-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/cf2 2023-04-12 02:54:23,037 DEBUG [StoreOpener-1c6b8f18205322cbb248ea4fb10155d8-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/cf2 2023-04-12 02:54:23,037 INFO [StoreOpener-1c6b8f18205322cbb248ea4fb10155d8-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1c6b8f18205322cbb248ea4fb10155d8 columnFamilyName cf2 2023-04-12 02:54:23,038 INFO [StoreOpener-1c6b8f18205322cbb248ea4fb10155d8-1] regionserver.HStore(310): Store=1c6b8f18205322cbb248ea4fb10155d8/cf2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-12 02:54:23,039 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8 2023-04-12 02:54:23,039 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8 2023-04-12 02:54:23,042 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testSplitTableRegionDeletedRowsDaughter descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-04-12 02:54:23,043 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 1c6b8f18205322cbb248ea4fb10155d8 2023-04-12 02:54:23,046 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-12 02:54:23,046 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 1c6b8f18205322cbb248ea4fb10155d8; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=60916410, jitterRate=-0.0922747552394867}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-04-12 02:54:23,046 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 1c6b8f18205322cbb248ea4fb10155d8: 2023-04-12 02:54:23,047 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testSplitTableRegionDeletedRowsDaughter,,1681268062677.1c6b8f18205322cbb248ea4fb10155d8., pid=71, masterSystemTime=1681268063024 2023-04-12 02:54:23,049 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testSplitTableRegionDeletedRowsDaughter,,1681268062677.1c6b8f18205322cbb248ea4fb10155d8. 2023-04-12 02:54:23,049 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testSplitTableRegionDeletedRowsDaughter,,1681268062677.1c6b8f18205322cbb248ea4fb10155d8. 2023-04-12 02:54:23,049 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=70 updating hbase:meta row=1c6b8f18205322cbb248ea4fb10155d8, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:23,049 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testSplitTableRegionDeletedRowsDaughter,,1681268062677.1c6b8f18205322cbb248ea4fb10155d8.","families":{"info":[{"qualifier":"regioninfo","vlen":73,"tag":[],"timestamp":"1681268063049"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1681268063049"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681268063049"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681268063049"}]},"ts":"1681268063049"} 2023-04-12 02:54:23,052 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=71, resume processing ppid=70 2023-04-12 02:54:23,052 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=71, ppid=70, state=SUCCESS; OpenRegionProcedure 1c6b8f18205322cbb248ea4fb10155d8, server=jenkins-hbase4.apache.org,34579,1681268031719 in 178 msec 2023-04-12 02:54:23,054 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=70, resume processing ppid=69 2023-04-12 02:54:23,054 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=70, ppid=69, state=SUCCESS; TransitRegionStateProcedure table=testSplitTableRegionDeletedRowsDaughter, region=1c6b8f18205322cbb248ea4fb10155d8, ASSIGN in 335 msec 2023-04-12 02:54:23,054 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=69, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=testSplitTableRegionDeletedRowsDaughter execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-12 02:54:23,054 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testSplitTableRegionDeletedRowsDaughter","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681268063054"}]},"ts":"1681268063054"} 2023-04-12 02:54:23,055 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testSplitTableRegionDeletedRowsDaughter, state=ENABLED in hbase:meta 2023-04-12 02:54:23,058 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=69, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=testSplitTableRegionDeletedRowsDaughter execute state=CREATE_TABLE_POST_OPERATION 2023-04-12 02:54:23,059 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=69, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionDeletedRowsDaughter in 381 msec 2023-04-12 02:54:23,186 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 02:54:23,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] procedure.ProcedureCoordinator(165): Submitting procedure testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,187 INFO [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testSplitTableRegionDeletedRowsDaughter' 2023-04-12 02:54:23,187 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:54:23,187 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testSplitTableRegionDeletedRowsDaughter' starting 'acquire' 2023-04-12 02:54:23,187 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testSplitTableRegionDeletedRowsDaughter', kicking off acquire phase on members. 2023-04-12 02:54:23,188 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,188 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,189 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:23,189 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:54:23,189 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:54:23,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:54:23,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:54:23,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:54:23,189 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:54:23,189 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:23,189 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 02:54:23,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:54:23,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:54:23,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:54:23,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,190 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,190 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,190 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 02:54:23,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 02:54:23,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 02:54:23,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,190 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,190 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,190 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,191 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,191 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionDeletedRowsDaughter' with timeout 60000ms 2023-04-12 02:54:23,191 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:54:23,191 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,191 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,191 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionDeletedRowsDaughter' with timeout 60000ms 2023-04-12 02:54:23,191 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionDeletedRowsDaughter' with timeout 60000ms 2023-04-12 02:54:23,191 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:54:23,191 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' starting 'acquire' stage 2023-04-12 02:54:23,191 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' starting 'acquire' stage 2023-04-12 02:54:23,191 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally acquired 2023-04-12 02:54:23,191 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:54:23,191 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43091,1681268031804' joining acquired barrier for procedure (testSplitTableRegionDeletedRowsDaughter) in zk 2023-04-12 02:54:23,191 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 02:54:23,191 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 02:54:23,191 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' starting 'acquire' stage 2023-04-12 02:54:23,191 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally acquired 2023-04-12 02:54:23,191 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44679,1681268031772' joining acquired barrier for procedure (testSplitTableRegionDeletedRowsDaughter) in zk 2023-04-12 02:54:23,191 DEBUG [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testSplitTableRegionDeletedRowsDaughter,,1681268062677.1c6b8f18205322cbb248ea4fb10155d8. 2023-04-12 02:54:23,191 DEBUG [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testSplitTableRegionDeletedRowsDaughter,,1681268062677.1c6b8f18205322cbb248ea4fb10155d8. started... 2023-04-12 02:54:23,192 INFO [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1c6b8f18205322cbb248ea4fb10155d8 2/2 column families, dataSize=120 B heapSize=928 B 2023-04-12 02:54:23,194 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,195 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,195 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,195 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 02:54:23,195 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,195 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 02:54:23,202 INFO [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=60 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/.tmp/cf1/789e1a943f304f83be48e491529ca4ad 2023-04-12 02:54:23,222 INFO [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=60 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/.tmp/cf2/926ca93402364365a7ac53b2e60b9bf5 2023-04-12 02:54:23,227 DEBUG [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/.tmp/cf1/789e1a943f304f83be48e491529ca4ad as hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/cf1/789e1a943f304f83be48e491529ca4ad 2023-04-12 02:54:23,231 INFO [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/cf1/789e1a943f304f83be48e491529ca4ad, entries=2, sequenceid=6, filesize=4.8 K 2023-04-12 02:54:23,232 DEBUG [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/.tmp/cf2/926ca93402364365a7ac53b2e60b9bf5 as hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/cf2/926ca93402364365a7ac53b2e60b9bf5 2023-04-12 02:54:23,236 INFO [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/cf2/926ca93402364365a7ac53b2e60b9bf5, entries=2, sequenceid=6, filesize=4.8 K 2023-04-12 02:54:23,237 INFO [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~120 B/120, heapSize ~896 B/896, currentSize=0 B/0 for 1c6b8f18205322cbb248ea4fb10155d8 in 45ms, sequenceid=6, compaction requested=false 2023-04-12 02:54:23,237 DEBUG [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'testSplitTableRegionDeletedRowsDaughter' 2023-04-12 02:54:23,238 DEBUG [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1c6b8f18205322cbb248ea4fb10155d8: 2023-04-12 02:54:23,238 DEBUG [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testSplitTableRegionDeletedRowsDaughter,,1681268062677.1c6b8f18205322cbb248ea4fb10155d8. 2023-04-12 02:54:23,238 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 02:54:23,238 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 02:54:23,238 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:23,238 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally acquired 2023-04-12 02:54:23,238 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34579,1681268031719' joining acquired barrier for procedure (testSplitTableRegionDeletedRowsDaughter) in zk 2023-04-12 02:54:23,241 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:23,241 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:23,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 02:54:23,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 02:54:23,241 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,241 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 02:54:23,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 02:54:23,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 02:54:23,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:23,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:23,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:23,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 02:54:23,244 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34579,1681268031719' joining acquired barrier for procedure 'testSplitTableRegionDeletedRowsDaughter' on coordinator 2023-04-12 02:54:23,244 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@598507fa[Count = 0] remaining members to acquire global barrier 2023-04-12 02:54:23,244 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testSplitTableRegionDeletedRowsDaughter' starting 'in-barrier' execution. 2023-04-12 02:54:23,244 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,246 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,246 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,246 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,246 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,246 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,246 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,246 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' received 'reached' from coordinator. 2023-04-12 02:54:23,246 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' received 'reached' from coordinator. 2023-04-12 02:54:23,246 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally completed 2023-04-12 02:54:23,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,246 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionDeletedRowsDaughter' completed for member 'jenkins-hbase4.apache.org,43091,1681268031804' in zk 2023-04-12 02:54:23,246 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally completed 2023-04-12 02:54:23,246 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionDeletedRowsDaughter' completed for member 'jenkins-hbase4.apache.org,44679,1681268031772' in zk 2023-04-12 02:54:23,246 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' received 'reached' from coordinator. 2023-04-12 02:54:23,246 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally completed 2023-04-12 02:54:23,246 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionDeletedRowsDaughter' completed for member 'jenkins-hbase4.apache.org,34579,1681268031719' in zk 2023-04-12 02:54:23,249 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' has notified controller of completion 2023-04-12 02:54:23,249 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:54:23,249 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' completed. 2023-04-12 02:54:23,249 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' has notified controller of completion 2023-04-12 02:54:23,250 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' has notified controller of completion 2023-04-12 02:54:23,250 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:54:23,250 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:54:23,250 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' completed. 2023-04-12 02:54:23,250 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' completed. 2023-04-12 02:54:23,250 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:23,251 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34579,1681268031719' released barrier for procedure'testSplitTableRegionDeletedRowsDaughter', counting down latch. Waiting for 0 more 2023-04-12 02:54:23,251 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 02:54:23,251 INFO [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testSplitTableRegionDeletedRowsDaughter' execution completed 2023-04-12 02:54:23,251 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 02:54:23,251 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 02:54:23,251 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,251 INFO [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testSplitTableRegionDeletedRowsDaughterincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 02:54:23,253 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,253 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,253 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,253 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,253 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 02:54:23,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 02:54:23,253 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:54:23,253 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,254 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:54:23,254 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,253 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:54:23,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,254 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:54:23,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:54:23,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 02:54:23,254 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:54:23,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:54:23,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,255 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:54:23,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:54:23,255 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:23,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,255 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:23,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 02:54:23,256 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:23,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:23,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:23,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:23,257 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:23,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 02:54:23,258 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:23,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,258 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:23,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:23,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:23,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:23,266 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:54:23,267 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:54:23,266 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:23,267 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:54:23,267 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:54:23,267 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:54:23,267 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:54:23,267 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,267 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:54:23,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:54:23,267 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:54:23,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:54:23,267 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:54:23,267 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:23,267 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:23,267 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testSplitTableRegionDeletedRowsDaughter' 2023-04-12 02:54:23,267 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 02:54:23,267 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:54:23,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:54:23,267 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,267 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:23,267 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,267 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:23,268 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:23,268 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:54:23,268 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:54:23,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:54:23,268 DEBUG [Listener at localhost/46667] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testSplitTableRegionDeletedRowsDaughter'' to complete. (max 20000 ms per retry) 2023-04-12 02:54:23,268 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:54:23,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:54:23,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:54:23,268 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,268 DEBUG [Listener at localhost/46667] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 02:54:23,268 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,368 DEBUG [Listener at localhost/46667] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 02:54:23,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 02:54:23,381 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 02:54:23,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] procedure.ProcedureCoordinator(143): Procedure testSplitTableRegionDeletedRowsDaughter was in running list but was completed. Accepting new attempt. 2023-04-12 02:54:23,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] procedure.ProcedureCoordinator(165): Submitting procedure testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,383 INFO [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testSplitTableRegionDeletedRowsDaughter' 2023-04-12 02:54:23,383 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:54:23,383 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testSplitTableRegionDeletedRowsDaughter' starting 'acquire' 2023-04-12 02:54:23,383 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testSplitTableRegionDeletedRowsDaughter', kicking off acquire phase on members. 2023-04-12 02:54:23,383 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,383 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,385 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:54:23,385 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:54:23,385 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:23,385 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:54:23,385 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:54:23,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:54:23,385 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:54:23,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:54:23,385 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:54:23,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:54:23,385 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:23,385 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 02:54:23,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,386 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,386 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,386 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 02:54:23,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,386 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 02:54:23,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,386 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 02:54:23,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,386 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionDeletedRowsDaughter is still present, removing 2023-04-12 02:54:23,386 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,387 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionDeletedRowsDaughter is still present, removing 2023-04-12 02:54:23,387 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,387 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,387 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionDeletedRowsDaughter' with timeout 60000ms 2023-04-12 02:54:23,387 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:54:23,387 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionDeletedRowsDaughter' with timeout 60000ms 2023-04-12 02:54:23,387 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:54:23,387 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' starting 'acquire' stage 2023-04-12 02:54:23,387 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' starting 'acquire' stage 2023-04-12 02:54:23,387 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally acquired 2023-04-12 02:54:23,387 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43091,1681268031804' joining acquired barrier for procedure (testSplitTableRegionDeletedRowsDaughter) in zk 2023-04-12 02:54:23,387 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 02:54:23,387 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionDeletedRowsDaughter is still present, removing 2023-04-12 02:54:23,387 DEBUG [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testSplitTableRegionDeletedRowsDaughter,,1681268062677.1c6b8f18205322cbb248ea4fb10155d8. 2023-04-12 02:54:23,387 DEBUG [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testSplitTableRegionDeletedRowsDaughter,,1681268062677.1c6b8f18205322cbb248ea4fb10155d8. started... 2023-04-12 02:54:23,387 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 02:54:23,387 INFO [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1c6b8f18205322cbb248ea4fb10155d8 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-04-12 02:54:23,387 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,387 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionDeletedRowsDaughter' with timeout 60000ms 2023-04-12 02:54:23,388 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:54:23,388 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' starting 'acquire' stage 2023-04-12 02:54:23,388 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally acquired 2023-04-12 02:54:23,388 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44679,1681268031772' joining acquired barrier for procedure (testSplitTableRegionDeletedRowsDaughter) in zk 2023-04-12 02:54:23,388 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,389 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,389 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,389 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 02:54:23,390 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,390 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 02:54:23,406 INFO [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=19 (bloomFilter=true), to=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/.tmp/cf1/0fd593a3724149b8ad466c2cef9ba846 2023-04-12 02:54:23,418 INFO [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=19 (bloomFilter=true), to=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/.tmp/cf2/77b0532593204cbd90904f2bbec317e7 2023-04-12 02:54:23,422 DEBUG [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/.tmp/cf1/0fd593a3724149b8ad466c2cef9ba846 as hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/cf1/0fd593a3724149b8ad466c2cef9ba846 2023-04-12 02:54:23,426 INFO [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/cf1/0fd593a3724149b8ad466c2cef9ba846, entries=10, sequenceid=19, filesize=5.0 K 2023-04-12 02:54:23,427 DEBUG [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/.tmp/cf2/77b0532593204cbd90904f2bbec317e7 as hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/cf2/77b0532593204cbd90904f2bbec317e7 2023-04-12 02:54:23,431 INFO [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/cf2/77b0532593204cbd90904f2bbec317e7, entries=10, sequenceid=19, filesize=5.0 K 2023-04-12 02:54:23,432 INFO [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for 1c6b8f18205322cbb248ea4fb10155d8 in 45ms, sequenceid=19, compaction requested=false 2023-04-12 02:54:23,432 DEBUG [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1c6b8f18205322cbb248ea4fb10155d8: 2023-04-12 02:54:23,432 DEBUG [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testSplitTableRegionDeletedRowsDaughter,,1681268062677.1c6b8f18205322cbb248ea4fb10155d8. 2023-04-12 02:54:23,432 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 02:54:23,432 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 02:54:23,432 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:23,432 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally acquired 2023-04-12 02:54:23,432 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34579,1681268031719' joining acquired barrier for procedure (testSplitTableRegionDeletedRowsDaughter) in zk 2023-04-12 02:54:23,435 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:23,435 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:23,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 02:54:23,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 02:54:23,436 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 02:54:23,436 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 02:54:23,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 02:54:23,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:23,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:23,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:23,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 02:54:23,439 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34579,1681268031719' joining acquired barrier for procedure 'testSplitTableRegionDeletedRowsDaughter' on coordinator 2023-04-12 02:54:23,439 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@f7b255b[Count = 0] remaining members to acquire global barrier 2023-04-12 02:54:23,439 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testSplitTableRegionDeletedRowsDaughter' starting 'in-barrier' execution. 2023-04-12 02:54:23,439 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,441 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,441 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,441 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' received 'reached' from coordinator. 2023-04-12 02:54:23,441 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally completed 2023-04-12 02:54:23,441 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionDeletedRowsDaughter' completed for member 'jenkins-hbase4.apache.org,43091,1681268031804' in zk 2023-04-12 02:54:23,441 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:23,441 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 02:54:23,443 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' has notified controller of completion 2023-04-12 02:54:23,443 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:54:23,443 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' completed. 2023-04-12 02:54:23,448 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,448 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,448 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,448 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,448 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' received 'reached' from coordinator. 2023-04-12 02:54:23,448 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally completed 2023-04-12 02:54:23,448 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionDeletedRowsDaughter' completed for member 'jenkins-hbase4.apache.org,34579,1681268031719' in zk 2023-04-12 02:54:23,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,448 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' received 'reached' from coordinator. 2023-04-12 02:54:23,448 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally completed 2023-04-12 02:54:23,449 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionDeletedRowsDaughter' completed for member 'jenkins-hbase4.apache.org,44679,1681268031772' in zk 2023-04-12 02:54:23,450 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' has notified controller of completion 2023-04-12 02:54:23,450 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:54:23,450 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' completed. 2023-04-12 02:54:23,450 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:23,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:23,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 02:54:23,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 02:54:23,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 02:54:23,452 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' has notified controller of completion 2023-04-12 02:54:23,452 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:54:23,452 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' completed. 2023-04-12 02:54:23,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 02:54:23,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:23,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:23,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:23,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 02:54:23,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:23,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:23,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:23,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testSplitTableRegionDeletedRowsDaughter' member 'jenkins-hbase4.apache.org,34579,1681268031719': 2023-04-12 02:54:23,456 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34579,1681268031719' released barrier for procedure'testSplitTableRegionDeletedRowsDaughter', counting down latch. Waiting for 0 more 2023-04-12 02:54:23,456 INFO [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testSplitTableRegionDeletedRowsDaughter' execution completed 2023-04-12 02:54:23,456 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 02:54:23,456 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 02:54:23,456 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,456 INFO [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testSplitTableRegionDeletedRowsDaughterincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 02:54:23,458 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,458 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:54:23,458 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,458 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,458 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,458 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,458 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,458 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:54:23,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 02:54:23,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 02:54:23,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,458 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,459 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:54:23,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:54:23,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,458 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:54:23,460 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:54:23,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:54:23,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 02:54:23,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,460 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:54:23,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:54:23,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,461 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:23,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,461 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:23,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 02:54:23,461 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:23,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:23,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:23,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:23,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 02:54:23,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,463 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:23,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:23,463 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:23,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:23,464 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:23,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:23,466 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:23,466 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,466 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:23,466 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:54:23,466 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:54:23,466 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:54:23,466 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:54:23,466 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:54:23,466 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:54:23,466 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:54:23,466 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:23,466 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:54:23,467 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,467 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:54:23,467 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:23,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:54:23,466 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testSplitTableRegionDeletedRowsDaughter' 2023-04-12 02:54:23,467 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 02:54:23,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:54:23,467 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:54:23,467 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:54:23,467 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:54:23,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:54:23,467 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:23,467 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:23,467 DEBUG [Listener at localhost/46667] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testSplitTableRegionDeletedRowsDaughter'' to complete. (max 20000 ms per retry) 2023-04-12 02:54:23,467 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:54:23,467 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:54:23,467 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,467 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:54:23,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:54:23,467 DEBUG [Listener at localhost/46667] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 02:54:23,568 DEBUG [Listener at localhost/46667] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 02:54:23,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 02:54:23,582 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 02:54:23,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] procedure.ProcedureCoordinator(143): Procedure testSplitTableRegionDeletedRowsDaughter was in running list but was completed. Accepting new attempt. 2023-04-12 02:54:23,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] procedure.ProcedureCoordinator(165): Submitting procedure testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,583 INFO [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testSplitTableRegionDeletedRowsDaughter' 2023-04-12 02:54:23,583 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:54:23,583 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testSplitTableRegionDeletedRowsDaughter' starting 'acquire' 2023-04-12 02:54:23,583 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testSplitTableRegionDeletedRowsDaughter', kicking off acquire phase on members. 2023-04-12 02:54:23,584 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,584 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,585 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:54:23,585 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:54:23,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:54:23,586 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:54:23,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:54:23,585 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:23,585 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:54:23,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:54:23,586 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:54:23,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:54:23,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,586 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:23,586 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 02:54:23,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,586 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,587 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,587 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 02:54:23,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,587 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,587 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionDeletedRowsDaughter is still present, removing 2023-04-12 02:54:23,587 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,587 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionDeletedRowsDaughter' with timeout 60000ms 2023-04-12 02:54:23,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 02:54:23,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 02:54:23,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,587 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:54:23,587 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,587 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,587 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' starting 'acquire' stage 2023-04-12 02:54:23,588 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionDeletedRowsDaughter is still present, removing 2023-04-12 02:54:23,588 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,588 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionDeletedRowsDaughter' with timeout 60000ms 2023-04-12 02:54:23,588 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:54:23,588 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionDeletedRowsDaughter is still present, removing 2023-04-12 02:54:23,588 DEBUG [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testSplitTableRegionDeletedRowsDaughter,,1681268062677.1c6b8f18205322cbb248ea4fb10155d8. 2023-04-12 02:54:23,588 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' starting 'acquire' stage 2023-04-12 02:54:23,588 DEBUG [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testSplitTableRegionDeletedRowsDaughter,,1681268062677.1c6b8f18205322cbb248ea4fb10155d8. started... 2023-04-12 02:54:23,588 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 02:54:23,588 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally acquired 2023-04-12 02:54:23,588 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,588 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44679,1681268031772' joining acquired barrier for procedure (testSplitTableRegionDeletedRowsDaughter) in zk 2023-04-12 02:54:23,588 INFO [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1c6b8f18205322cbb248ea4fb10155d8 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-04-12 02:54:23,588 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 02:54:23,588 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionDeletedRowsDaughter' with timeout 60000ms 2023-04-12 02:54:23,588 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:54:23,588 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' starting 'acquire' stage 2023-04-12 02:54:23,588 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally acquired 2023-04-12 02:54:23,588 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43091,1681268031804' joining acquired barrier for procedure (testSplitTableRegionDeletedRowsDaughter) in zk 2023-04-12 02:54:23,595 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,596 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,596 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,596 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 02:54:23,596 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,596 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 02:54:23,606 INFO [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=32 (bloomFilter=true), to=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/.tmp/cf1/bce8056f97d34999b5a16bbacddae4f5 2023-04-12 02:54:23,624 INFO [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=32 (bloomFilter=true), to=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/.tmp/cf2/381b6116e3b64f31a2b235e9f7067d52 2023-04-12 02:54:23,629 DEBUG [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/.tmp/cf1/bce8056f97d34999b5a16bbacddae4f5 as hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/cf1/bce8056f97d34999b5a16bbacddae4f5 2023-04-12 02:54:23,633 INFO [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/cf1/bce8056f97d34999b5a16bbacddae4f5, entries=10, sequenceid=32, filesize=5.0 K 2023-04-12 02:54:23,634 DEBUG [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/.tmp/cf2/381b6116e3b64f31a2b235e9f7067d52 as hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/cf2/381b6116e3b64f31a2b235e9f7067d52 2023-04-12 02:54:23,638 INFO [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/cf2/381b6116e3b64f31a2b235e9f7067d52, entries=10, sequenceid=32, filesize=5.0 K 2023-04-12 02:54:23,639 INFO [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for 1c6b8f18205322cbb248ea4fb10155d8 in 51ms, sequenceid=32, compaction requested=true 2023-04-12 02:54:23,640 DEBUG [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1c6b8f18205322cbb248ea4fb10155d8: 2023-04-12 02:54:23,640 DEBUG [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testSplitTableRegionDeletedRowsDaughter,,1681268062677.1c6b8f18205322cbb248ea4fb10155d8. 2023-04-12 02:54:23,640 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 02:54:23,640 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 02:54:23,640 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:23,640 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally acquired 2023-04-12 02:54:23,640 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34579,1681268031719' joining acquired barrier for procedure (testSplitTableRegionDeletedRowsDaughter) in zk 2023-04-12 02:54:23,644 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:23,644 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:23,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 02:54:23,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 02:54:23,645 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,645 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 02:54:23,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 02:54:23,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 02:54:23,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:23,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:23,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:23,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 02:54:23,647 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34579,1681268031719' joining acquired barrier for procedure 'testSplitTableRegionDeletedRowsDaughter' on coordinator 2023-04-12 02:54:23,647 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@730144b3[Count = 0] remaining members to acquire global barrier 2023-04-12 02:54:23,647 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testSplitTableRegionDeletedRowsDaughter' starting 'in-barrier' execution. 2023-04-12 02:54:23,647 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,648 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,648 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,648 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,648 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,648 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,648 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:23,648 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 02:54:23,648 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,648 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' received 'reached' from coordinator. 2023-04-12 02:54:23,648 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally completed 2023-04-12 02:54:23,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,648 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionDeletedRowsDaughter' completed for member 'jenkins-hbase4.apache.org,43091,1681268031804' in zk 2023-04-12 02:54:23,649 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' received 'reached' from coordinator. 2023-04-12 02:54:23,649 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally completed 2023-04-12 02:54:23,649 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionDeletedRowsDaughter' completed for member 'jenkins-hbase4.apache.org,44679,1681268031772' in zk 2023-04-12 02:54:23,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,649 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' received 'reached' from coordinator. 2023-04-12 02:54:23,649 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally completed 2023-04-12 02:54:23,649 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionDeletedRowsDaughter' completed for member 'jenkins-hbase4.apache.org,34579,1681268031719' in zk 2023-04-12 02:54:23,650 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' has notified controller of completion 2023-04-12 02:54:23,650 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:54:23,650 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' completed. 2023-04-12 02:54:23,652 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' has notified controller of completion 2023-04-12 02:54:23,652 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:23,652 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' has notified controller of completion 2023-04-12 02:54:23,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:23,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 02:54:23,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 02:54:23,652 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:54:23,652 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:54:23,652 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' completed. 2023-04-12 02:54:23,653 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' completed. 2023-04-12 02:54:23,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 02:54:23,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 02:54:23,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:23,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:23,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:23,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 02:54:23,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:23,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:23,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:23,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testSplitTableRegionDeletedRowsDaughter' member 'jenkins-hbase4.apache.org,34579,1681268031719': 2023-04-12 02:54:23,655 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34579,1681268031719' released barrier for procedure'testSplitTableRegionDeletedRowsDaughter', counting down latch. Waiting for 0 more 2023-04-12 02:54:23,655 INFO [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testSplitTableRegionDeletedRowsDaughter' execution completed 2023-04-12 02:54:23,655 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 02:54:23,655 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 02:54:23,655 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,655 INFO [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testSplitTableRegionDeletedRowsDaughterincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 02:54:23,657 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,657 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,657 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,657 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,657 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:54:23,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 02:54:23,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 02:54:23,657 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:54:23,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,657 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:54:23,658 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:54:23,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:54:23,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 02:54:23,658 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:54:23,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:54:23,658 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:23,658 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:54:23,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:54:23,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,658 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:23,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 02:54:23,659 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:23,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:23,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:23,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:23,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 02:54:23,661 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:23,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,661 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:23,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:23,661 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:23,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:23,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:23,663 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:54:23,663 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:23,663 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:54:23,664 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:54:23,663 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:54:23,663 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:54:23,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:54:23,664 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:54:23,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:54:23,664 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testSplitTableRegionDeletedRowsDaughter' 2023-04-12 02:54:23,664 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 02:54:23,664 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:54:23,664 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:54:23,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:54:23,664 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,664 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:54:23,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:54:23,664 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:54:23,664 DEBUG [Listener at localhost/46667] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testSplitTableRegionDeletedRowsDaughter'' to complete. (max 20000 ms per retry) 2023-04-12 02:54:23,664 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:54:23,664 DEBUG [Listener at localhost/46667] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 02:54:23,664 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:23,665 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:54:23,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:54:23,665 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:23,664 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:54:23,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:54:23,665 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,665 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:23,665 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,665 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:23,665 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:23,665 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,665 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,765 DEBUG [Listener at localhost/46667] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 02:54:23,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 02:54:23,779 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 02:54:23,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] procedure.ProcedureCoordinator(143): Procedure testSplitTableRegionDeletedRowsDaughter was in running list but was completed. Accepting new attempt. 2023-04-12 02:54:23,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] procedure.ProcedureCoordinator(165): Submitting procedure testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,780 INFO [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testSplitTableRegionDeletedRowsDaughter' 2023-04-12 02:54:23,781 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:54:23,781 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testSplitTableRegionDeletedRowsDaughter' starting 'acquire' 2023-04-12 02:54:23,781 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testSplitTableRegionDeletedRowsDaughter', kicking off acquire phase on members. 2023-04-12 02:54:23,781 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,781 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,783 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:54:23,783 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:54:23,783 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:54:23,783 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:54:23,783 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:23,783 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:54:23,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:54:23,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:54:23,783 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:54:23,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:54:23,783 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:23,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,783 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 02:54:23,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,784 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,784 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,784 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 02:54:23,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 02:54:23,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,784 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 02:54:23,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,784 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,784 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,785 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionDeletedRowsDaughter is still present, removing 2023-04-12 02:54:23,785 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,785 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionDeletedRowsDaughter' with timeout 60000ms 2023-04-12 02:54:23,785 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:54:23,785 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionDeletedRowsDaughter is still present, removing 2023-04-12 02:54:23,785 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' starting 'acquire' stage 2023-04-12 02:54:23,785 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionDeletedRowsDaughter is still present, removing 2023-04-12 02:54:23,785 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally acquired 2023-04-12 02:54:23,785 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,785 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43091,1681268031804' joining acquired barrier for procedure (testSplitTableRegionDeletedRowsDaughter) in zk 2023-04-12 02:54:23,785 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionDeletedRowsDaughter' with timeout 60000ms 2023-04-12 02:54:23,785 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,785 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:54:23,785 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionDeletedRowsDaughter' with timeout 60000ms 2023-04-12 02:54:23,785 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:54:23,785 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' starting 'acquire' stage 2023-04-12 02:54:23,785 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally acquired 2023-04-12 02:54:23,785 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' starting 'acquire' stage 2023-04-12 02:54:23,786 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44679,1681268031772' joining acquired barrier for procedure (testSplitTableRegionDeletedRowsDaughter) in zk 2023-04-12 02:54:23,786 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 02:54:23,786 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 02:54:23,786 DEBUG [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testSplitTableRegionDeletedRowsDaughter,,1681268062677.1c6b8f18205322cbb248ea4fb10155d8. 2023-04-12 02:54:23,786 DEBUG [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testSplitTableRegionDeletedRowsDaughter,,1681268062677.1c6b8f18205322cbb248ea4fb10155d8. started... 2023-04-12 02:54:23,786 INFO [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1c6b8f18205322cbb248ea4fb10155d8 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-04-12 02:54:23,788 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,789 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,789 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,789 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 02:54:23,789 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,789 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 02:54:23,796 INFO [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=45 (bloomFilter=true), to=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/.tmp/cf1/985388b30d784536a1f270feb7f52366 2023-04-12 02:54:23,814 INFO [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=45 (bloomFilter=true), to=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/.tmp/cf2/ff9f5d4721ca4fc5a30cfe3141f45ca7 2023-04-12 02:54:23,818 DEBUG [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/.tmp/cf1/985388b30d784536a1f270feb7f52366 as hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/cf1/985388b30d784536a1f270feb7f52366 2023-04-12 02:54:23,823 INFO [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/cf1/985388b30d784536a1f270feb7f52366, entries=10, sequenceid=45, filesize=5.0 K 2023-04-12 02:54:23,824 DEBUG [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/.tmp/cf2/ff9f5d4721ca4fc5a30cfe3141f45ca7 as hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/cf2/ff9f5d4721ca4fc5a30cfe3141f45ca7 2023-04-12 02:54:23,828 INFO [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/cf2/ff9f5d4721ca4fc5a30cfe3141f45ca7, entries=10, sequenceid=45, filesize=5.0 K 2023-04-12 02:54:23,830 INFO [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for 1c6b8f18205322cbb248ea4fb10155d8 in 44ms, sequenceid=45, compaction requested=true 2023-04-12 02:54:23,830 DEBUG [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1c6b8f18205322cbb248ea4fb10155d8: 2023-04-12 02:54:23,830 DEBUG [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testSplitTableRegionDeletedRowsDaughter,,1681268062677.1c6b8f18205322cbb248ea4fb10155d8. 2023-04-12 02:54:23,830 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 02:54:23,830 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 02:54:23,830 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:23,830 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally acquired 2023-04-12 02:54:23,830 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34579,1681268031719' joining acquired barrier for procedure (testSplitTableRegionDeletedRowsDaughter) in zk 2023-04-12 02:54:23,832 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:23,832 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:23,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 02:54:23,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 02:54:23,832 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,832 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 02:54:23,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 02:54:23,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 02:54:23,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:23,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:23,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:23,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 02:54:23,834 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34579,1681268031719' joining acquired barrier for procedure 'testSplitTableRegionDeletedRowsDaughter' on coordinator 2023-04-12 02:54:23,834 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@673d6b0a[Count = 0] remaining members to acquire global barrier 2023-04-12 02:54:23,834 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testSplitTableRegionDeletedRowsDaughter' starting 'in-barrier' execution. 2023-04-12 02:54:23,835 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,836 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,836 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,836 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,836 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' received 'reached' from coordinator. 2023-04-12 02:54:23,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,836 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:23,836 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 02:54:23,836 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally completed 2023-04-12 02:54:23,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,836 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionDeletedRowsDaughter' completed for member 'jenkins-hbase4.apache.org,34579,1681268031719' in zk 2023-04-12 02:54:23,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,836 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' received 'reached' from coordinator. 2023-04-12 02:54:23,836 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally completed 2023-04-12 02:54:23,836 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' received 'reached' from coordinator. 2023-04-12 02:54:23,837 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally completed 2023-04-12 02:54:23,837 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionDeletedRowsDaughter' completed for member 'jenkins-hbase4.apache.org,44679,1681268031772' in zk 2023-04-12 02:54:23,837 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionDeletedRowsDaughter' completed for member 'jenkins-hbase4.apache.org,43091,1681268031804' in zk 2023-04-12 02:54:23,839 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:23,839 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' has notified controller of completion 2023-04-12 02:54:23,839 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:54:23,839 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' completed. 2023-04-12 02:54:23,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:23,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 02:54:23,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 02:54:23,840 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' has notified controller of completion 2023-04-12 02:54:23,840 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' has notified controller of completion 2023-04-12 02:54:23,840 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:54:23,840 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:54:23,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 02:54:23,840 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' completed. 2023-04-12 02:54:23,840 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' completed. 2023-04-12 02:54:23,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 02:54:23,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:23,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:23,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:23,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 02:54:23,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:23,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:23,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:23,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testSplitTableRegionDeletedRowsDaughter' member 'jenkins-hbase4.apache.org,34579,1681268031719': 2023-04-12 02:54:23,843 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34579,1681268031719' released barrier for procedure'testSplitTableRegionDeletedRowsDaughter', counting down latch. Waiting for 0 more 2023-04-12 02:54:23,843 INFO [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testSplitTableRegionDeletedRowsDaughter' execution completed 2023-04-12 02:54:23,843 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 02:54:23,844 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 02:54:23,844 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,844 INFO [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testSplitTableRegionDeletedRowsDaughterincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 02:54:23,848 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,848 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,848 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,848 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,848 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,848 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:54:23,848 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,848 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:54:23,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 02:54:23,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 02:54:23,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,848 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,848 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:54:23,849 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:23,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 02:54:23,849 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:54:23,849 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:54:23,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:54:23,849 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:54:23,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:54:23,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:54:23,849 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:23,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,850 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:23,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 02:54:23,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:23,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:23,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:23,851 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:23,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 02:54:23,851 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:23,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,852 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:23,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:23,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:23,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:23,854 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:23,854 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:54:23,854 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:54:23,854 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:54:23,854 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,854 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:54:23,854 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testSplitTableRegionDeletedRowsDaughter' 2023-04-12 02:54:23,855 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 02:54:23,854 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:23,854 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:54:23,855 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:23,854 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:54:23,854 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:54:23,854 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:54:23,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:54:23,854 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:54:23,854 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:54:23,855 DEBUG [Listener at localhost/46667] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testSplitTableRegionDeletedRowsDaughter'' to complete. (max 20000 ms per retry) 2023-04-12 02:54:23,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:54:23,855 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:54:23,855 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:54:23,855 DEBUG [Listener at localhost/46667] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 02:54:23,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:54:23,855 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:54:23,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:54:23,855 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:23,855 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,856 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:23,856 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:23,856 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,856 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:54:23,856 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:54:23,955 DEBUG [Listener at localhost/46667] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 02:54:23,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 02:54:23,985 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 02:54:23,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] procedure.ProcedureCoordinator(143): Procedure testSplitTableRegionDeletedRowsDaughter was in running list but was completed. Accepting new attempt. 2023-04-12 02:54:23,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] procedure.ProcedureCoordinator(165): Submitting procedure testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,987 INFO [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testSplitTableRegionDeletedRowsDaughter' 2023-04-12 02:54:23,987 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:54:23,987 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testSplitTableRegionDeletedRowsDaughter' starting 'acquire' 2023-04-12 02:54:23,987 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testSplitTableRegionDeletedRowsDaughter', kicking off acquire phase on members. 2023-04-12 02:54:23,988 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,988 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,989 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:54:23,989 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:54:23,989 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:54:23,989 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:54:23,989 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:54:23,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:54:23,989 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:23,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:54:23,989 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:54:23,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:54:23,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,990 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:23,991 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,991 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 02:54:23,991 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 02:54:23,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,991 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 02:54:23,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,991 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,992 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 02:54:23,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:23,993 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,000 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionDeletedRowsDaughter is still present, removing 2023-04-12 02:54:24,000 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,000 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionDeletedRowsDaughter' with timeout 60000ms 2023-04-12 02:54:24,000 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:54:24,000 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' starting 'acquire' stage 2023-04-12 02:54:24,000 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally acquired 2023-04-12 02:54:24,000 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43091,1681268031804' joining acquired barrier for procedure (testSplitTableRegionDeletedRowsDaughter) in zk 2023-04-12 02:54:24,002 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,003 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,003 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 02:54:24,004 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionDeletedRowsDaughter is still present, removing 2023-04-12 02:54:24,004 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,004 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionDeletedRowsDaughter' with timeout 60000ms 2023-04-12 02:54:24,004 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:54:24,004 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' starting 'acquire' stage 2023-04-12 02:54:24,004 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally acquired 2023-04-12 02:54:24,004 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44679,1681268031772' joining acquired barrier for procedure (testSplitTableRegionDeletedRowsDaughter) in zk 2023-04-12 02:54:24,006 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,008 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,008 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 02:54:24,009 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionDeletedRowsDaughter is still present, removing 2023-04-12 02:54:24,009 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,009 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionDeletedRowsDaughter' with timeout 60000ms 2023-04-12 02:54:24,009 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:54:24,009 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' starting 'acquire' stage 2023-04-12 02:54:24,011 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 02:54:24,011 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 02:54:24,011 DEBUG [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testSplitTableRegionDeletedRowsDaughter,,1681268062677.1c6b8f18205322cbb248ea4fb10155d8. 2023-04-12 02:54:24,011 DEBUG [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testSplitTableRegionDeletedRowsDaughter,,1681268062677.1c6b8f18205322cbb248ea4fb10155d8. started... 2023-04-12 02:54:24,011 INFO [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1c6b8f18205322cbb248ea4fb10155d8 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-04-12 02:54:24,057 INFO [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=58 (bloomFilter=true), to=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/.tmp/cf1/710424cef2c64d2ebc740c81294a95cb 2023-04-12 02:54:24,106 INFO [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=58 (bloomFilter=true), to=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/.tmp/cf2/85cb745e6ba6488f8367339705326ef9 2023-04-12 02:54:24,117 DEBUG [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/.tmp/cf1/710424cef2c64d2ebc740c81294a95cb as hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/cf1/710424cef2c64d2ebc740c81294a95cb 2023-04-12 02:54:24,123 INFO [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/cf1/710424cef2c64d2ebc740c81294a95cb, entries=10, sequenceid=58, filesize=5.0 K 2023-04-12 02:54:24,124 DEBUG [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/.tmp/cf2/85cb745e6ba6488f8367339705326ef9 as hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/cf2/85cb745e6ba6488f8367339705326ef9 2023-04-12 02:54:24,130 INFO [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/cf2/85cb745e6ba6488f8367339705326ef9, entries=10, sequenceid=58, filesize=5.0 K 2023-04-12 02:54:24,131 INFO [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for 1c6b8f18205322cbb248ea4fb10155d8 in 120ms, sequenceid=58, compaction requested=true 2023-04-12 02:54:24,131 DEBUG [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1c6b8f18205322cbb248ea4fb10155d8: 2023-04-12 02:54:24,131 DEBUG [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testSplitTableRegionDeletedRowsDaughter,,1681268062677.1c6b8f18205322cbb248ea4fb10155d8. 2023-04-12 02:54:24,131 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 02:54:24,131 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 02:54:24,131 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:24,131 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally acquired 2023-04-12 02:54:24,131 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34579,1681268031719' joining acquired barrier for procedure (testSplitTableRegionDeletedRowsDaughter) in zk 2023-04-12 02:54:24,135 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:24,135 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:24,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 02:54:24,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 02:54:24,136 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 02:54:24,136 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 02:54:24,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 02:54:24,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:24,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:24,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:24,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 02:54:24,138 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34579,1681268031719' joining acquired barrier for procedure 'testSplitTableRegionDeletedRowsDaughter' on coordinator 2023-04-12 02:54:24,138 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@16760ac0[Count = 0] remaining members to acquire global barrier 2023-04-12 02:54:24,138 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testSplitTableRegionDeletedRowsDaughter' starting 'in-barrier' execution. 2023-04-12 02:54:24,138 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,139 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,140 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,140 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,139 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,140 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,140 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,140 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' received 'reached' from coordinator. 2023-04-12 02:54:24,140 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally completed 2023-04-12 02:54:24,140 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionDeletedRowsDaughter' completed for member 'jenkins-hbase4.apache.org,43091,1681268031804' in zk 2023-04-12 02:54:24,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,140 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' received 'reached' from coordinator. 2023-04-12 02:54:24,140 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' received 'reached' from coordinator. 2023-04-12 02:54:24,140 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally completed 2023-04-12 02:54:24,140 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionDeletedRowsDaughter' completed for member 'jenkins-hbase4.apache.org,44679,1681268031772' in zk 2023-04-12 02:54:24,140 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally completed 2023-04-12 02:54:24,140 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionDeletedRowsDaughter' completed for member 'jenkins-hbase4.apache.org,34579,1681268031719' in zk 2023-04-12 02:54:24,141 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:24,141 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 02:54:24,143 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' has notified controller of completion 2023-04-12 02:54:24,143 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' has notified controller of completion 2023-04-12 02:54:24,143 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' has notified controller of completion 2023-04-12 02:54:24,143 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:24,143 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:54:24,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:24,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 02:54:24,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 02:54:24,143 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' completed. 2023-04-12 02:54:24,143 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:54:24,143 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:54:24,143 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' completed. 2023-04-12 02:54:24,143 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' completed. 2023-04-12 02:54:24,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 02:54:24,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 02:54:24,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:24,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:24,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:24,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 02:54:24,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:24,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:24,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:24,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testSplitTableRegionDeletedRowsDaughter' member 'jenkins-hbase4.apache.org,34579,1681268031719': 2023-04-12 02:54:24,150 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34579,1681268031719' released barrier for procedure'testSplitTableRegionDeletedRowsDaughter', counting down latch. Waiting for 0 more 2023-04-12 02:54:24,150 INFO [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testSplitTableRegionDeletedRowsDaughter' execution completed 2023-04-12 02:54:24,150 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 02:54:24,150 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 02:54:24,150 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,150 INFO [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testSplitTableRegionDeletedRowsDaughterincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 02:54:24,153 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,153 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,153 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 02:54:24,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 02:54:24,153 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,153 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:54:24,153 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,153 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,153 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:54:24,153 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,153 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:54:24,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,155 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:54:24,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:54:24,155 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:54:24,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:54:24,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 02:54:24,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:54:24,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:54:24,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,157 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:24,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 02:54:24,175 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:24,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,175 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:24,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:24,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:24,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:24,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 02:54:24,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,183 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:24,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:24,183 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:24,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:24,184 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:24,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:24,186 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:24,187 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:54:24,187 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,187 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:54:24,187 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:54:24,187 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:54:24,187 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:54:24,187 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:54:24,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:54:24,187 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testSplitTableRegionDeletedRowsDaughter' 2023-04-12 02:54:24,187 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 02:54:24,187 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:54:24,187 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:54:24,187 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:24,188 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:24,188 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,188 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:24,188 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,188 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:24,188 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:24,188 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,188 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,187 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:54:24,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:54:24,187 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:54:24,187 DEBUG [Listener at localhost/46667] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testSplitTableRegionDeletedRowsDaughter'' to complete. (max 20000 ms per retry) 2023-04-12 02:54:24,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:54:24,188 DEBUG [Listener at localhost/46667] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 02:54:24,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:54:24,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:54:24,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:54:24,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:54:24,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:54:24,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:54:24,288 DEBUG [Listener at localhost/46667] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 02:54:24,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 02:54:24,307 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 02:54:24,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] procedure.ProcedureCoordinator(143): Procedure testSplitTableRegionDeletedRowsDaughter was in running list but was completed. Accepting new attempt. 2023-04-12 02:54:24,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] procedure.ProcedureCoordinator(165): Submitting procedure testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,320 INFO [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testSplitTableRegionDeletedRowsDaughter' 2023-04-12 02:54:24,320 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:54:24,320 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testSplitTableRegionDeletedRowsDaughter' starting 'acquire' 2023-04-12 02:54:24,320 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testSplitTableRegionDeletedRowsDaughter', kicking off acquire phase on members. 2023-04-12 02:54:24,321 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,321 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,326 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:54:24,326 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:54:24,326 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:54:24,326 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:54:24,326 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:54:24,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:54:24,326 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:54:24,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:54:24,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:54:24,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,327 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,328 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 02:54:24,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,328 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,328 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 02:54:24,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,329 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 02:54:24,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,329 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,330 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:24,331 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:24,331 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 02:54:24,340 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionDeletedRowsDaughter is still present, removing 2023-04-12 02:54:24,340 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,340 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionDeletedRowsDaughter' with timeout 60000ms 2023-04-12 02:54:24,340 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:54:24,340 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' starting 'acquire' stage 2023-04-12 02:54:24,340 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally acquired 2023-04-12 02:54:24,340 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43091,1681268031804' joining acquired barrier for procedure (testSplitTableRegionDeletedRowsDaughter) in zk 2023-04-12 02:54:24,342 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,342 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,343 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 02:54:24,350 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionDeletedRowsDaughter is still present, removing 2023-04-12 02:54:24,350 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,351 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionDeletedRowsDaughter' with timeout 60000ms 2023-04-12 02:54:24,351 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:54:24,351 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' starting 'acquire' stage 2023-04-12 02:54:24,351 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally acquired 2023-04-12 02:54:24,351 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44679,1681268031772' joining acquired barrier for procedure (testSplitTableRegionDeletedRowsDaughter) in zk 2023-04-12 02:54:24,353 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionDeletedRowsDaughter is still present, removing 2023-04-12 02:54:24,353 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,353 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionDeletedRowsDaughter' with timeout 60000ms 2023-04-12 02:54:24,353 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:54:24,353 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' starting 'acquire' stage 2023-04-12 02:54:24,354 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,354 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,354 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 02:54:24,372 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 02:54:24,372 DEBUG [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testSplitTableRegionDeletedRowsDaughter,,1681268062677.1c6b8f18205322cbb248ea4fb10155d8. 2023-04-12 02:54:24,372 DEBUG [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testSplitTableRegionDeletedRowsDaughter,,1681268062677.1c6b8f18205322cbb248ea4fb10155d8. started... 2023-04-12 02:54:24,372 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 02:54:24,372 INFO [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1c6b8f18205322cbb248ea4fb10155d8 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-04-12 02:54:24,434 INFO [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=71 (bloomFilter=true), to=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/.tmp/cf1/9c4efc02782d420291d1fe480cade0d3 2023-04-12 02:54:24,482 INFO [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=71 (bloomFilter=true), to=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/.tmp/cf2/e6cbb96ab13a4b67a21da11e8efd233f 2023-04-12 02:54:24,489 DEBUG [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/.tmp/cf1/9c4efc02782d420291d1fe480cade0d3 as hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/cf1/9c4efc02782d420291d1fe480cade0d3 2023-04-12 02:54:24,494 INFO [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/cf1/9c4efc02782d420291d1fe480cade0d3, entries=10, sequenceid=71, filesize=5.0 K 2023-04-12 02:54:24,496 DEBUG [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/.tmp/cf2/e6cbb96ab13a4b67a21da11e8efd233f as hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/cf2/e6cbb96ab13a4b67a21da11e8efd233f 2023-04-12 02:54:24,502 INFO [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/cf2/e6cbb96ab13a4b67a21da11e8efd233f, entries=10, sequenceid=71, filesize=5.0 K 2023-04-12 02:54:24,503 INFO [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for 1c6b8f18205322cbb248ea4fb10155d8 in 131ms, sequenceid=71, compaction requested=true 2023-04-12 02:54:24,504 DEBUG [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1c6b8f18205322cbb248ea4fb10155d8: 2023-04-12 02:54:24,504 DEBUG [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testSplitTableRegionDeletedRowsDaughter,,1681268062677.1c6b8f18205322cbb248ea4fb10155d8. 2023-04-12 02:54:24,504 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 02:54:24,504 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 02:54:24,504 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:24,504 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally acquired 2023-04-12 02:54:24,504 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34579,1681268031719' joining acquired barrier for procedure (testSplitTableRegionDeletedRowsDaughter) in zk 2023-04-12 02:54:24,507 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:24,507 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:24,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 02:54:24,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 02:54:24,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 02:54:24,508 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,508 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 02:54:24,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 02:54:24,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:24,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:24,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:24,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 02:54:24,510 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34579,1681268031719' joining acquired barrier for procedure 'testSplitTableRegionDeletedRowsDaughter' on coordinator 2023-04-12 02:54:24,510 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@40f1b4a8[Count = 0] remaining members to acquire global barrier 2023-04-12 02:54:24,510 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testSplitTableRegionDeletedRowsDaughter' starting 'in-barrier' execution. 2023-04-12 02:54:24,510 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,512 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,512 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,512 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,512 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,512 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' received 'reached' from coordinator. 2023-04-12 02:54:24,512 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' received 'reached' from coordinator. 2023-04-12 02:54:24,512 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally completed 2023-04-12 02:54:24,512 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,512 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionDeletedRowsDaughter' completed for member 'jenkins-hbase4.apache.org,34579,1681268031719' in zk 2023-04-12 02:54:24,512 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally completed 2023-04-12 02:54:24,512 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionDeletedRowsDaughter' completed for member 'jenkins-hbase4.apache.org,43091,1681268031804' in zk 2023-04-12 02:54:24,512 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,513 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' received 'reached' from coordinator. 2023-04-12 02:54:24,513 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally completed 2023-04-12 02:54:24,513 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionDeletedRowsDaughter' completed for member 'jenkins-hbase4.apache.org,44679,1681268031772' in zk 2023-04-12 02:54:24,513 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:24,513 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 02:54:24,514 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:24,514 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' has notified controller of completion 2023-04-12 02:54:24,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:24,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 02:54:24,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 02:54:24,514 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:54:24,514 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' completed. 2023-04-12 02:54:24,515 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' has notified controller of completion 2023-04-12 02:54:24,515 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:54:24,515 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' completed. 2023-04-12 02:54:24,516 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' has notified controller of completion 2023-04-12 02:54:24,517 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:54:24,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 02:54:24,517 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' completed. 2023-04-12 02:54:24,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 02:54:24,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:24,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:24,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:24,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 02:54:24,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:24,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:24,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:24,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testSplitTableRegionDeletedRowsDaughter' member 'jenkins-hbase4.apache.org,34579,1681268031719': 2023-04-12 02:54:24,521 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34579,1681268031719' released barrier for procedure'testSplitTableRegionDeletedRowsDaughter', counting down latch. Waiting for 0 more 2023-04-12 02:54:24,521 INFO [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testSplitTableRegionDeletedRowsDaughter' execution completed 2023-04-12 02:54:24,521 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 02:54:24,521 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 02:54:24,521 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,521 INFO [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testSplitTableRegionDeletedRowsDaughterincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 02:54:24,522 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,522 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:54:24,522 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,523 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,523 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,523 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 02:54:24,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 02:54:24,523 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:54:24,523 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,523 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:54:24,522 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,523 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:54:24,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:54:24,523 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:54:24,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:54:24,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 02:54:24,523 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:24,524 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:54:24,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:54:24,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,524 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:24,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 02:54:24,525 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:24,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:24,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:24,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:24,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 02:54:24,527 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:24,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,527 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:24,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:24,527 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:24,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:24,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:24,530 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:54:24,530 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:54:24,531 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:54:24,531 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:54:24,531 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:54:24,531 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:54:24,531 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:54:24,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:54:24,531 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:54:24,531 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testSplitTableRegionDeletedRowsDaughter' 2023-04-12 02:54:24,531 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 02:54:24,531 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:54:24,531 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:54:24,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:54:24,531 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:54:24,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:54:24,531 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:24,531 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,531 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:24,531 DEBUG [Listener at localhost/46667] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testSplitTableRegionDeletedRowsDaughter'' to complete. (max 20000 ms per retry) 2023-04-12 02:54:24,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:54:24,532 DEBUG [Listener at localhost/46667] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 02:54:24,531 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:24,532 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,532 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:24,532 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,532 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:24,532 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:24,532 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,532 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,533 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:54:24,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:54:24,533 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:54:24,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:54:24,632 DEBUG [Listener at localhost/46667] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 02:54:24,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 02:54:24,647 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 02:54:24,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] procedure.ProcedureCoordinator(143): Procedure testSplitTableRegionDeletedRowsDaughter was in running list but was completed. Accepting new attempt. 2023-04-12 02:54:24,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] procedure.ProcedureCoordinator(165): Submitting procedure testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,655 INFO [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testSplitTableRegionDeletedRowsDaughter' 2023-04-12 02:54:24,655 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:54:24,655 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testSplitTableRegionDeletedRowsDaughter' starting 'acquire' 2023-04-12 02:54:24,656 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testSplitTableRegionDeletedRowsDaughter', kicking off acquire phase on members. 2023-04-12 02:54:24,656 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,656 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,658 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:54:24,658 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:54:24,658 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:24,658 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:54:24,658 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:54:24,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:54:24,658 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:54:24,658 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:54:24,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:54:24,658 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:24,658 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 02:54:24,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:54:24,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,659 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,659 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,659 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 02:54:24,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,659 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 02:54:24,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,660 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,660 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionDeletedRowsDaughter is still present, removing 2023-04-12 02:54:24,660 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,660 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionDeletedRowsDaughter is still present, removing 2023-04-12 02:54:24,660 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionDeletedRowsDaughter' with timeout 60000ms 2023-04-12 02:54:24,660 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 02:54:24,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,660 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:54:24,660 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionDeletedRowsDaughter' with timeout 60000ms 2023-04-12 02:54:24,660 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:54:24,660 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' starting 'acquire' stage 2023-04-12 02:54:24,660 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally acquired 2023-04-12 02:54:24,660 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,660 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43091,1681268031804' joining acquired barrier for procedure (testSplitTableRegionDeletedRowsDaughter) in zk 2023-04-12 02:54:24,660 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' starting 'acquire' stage 2023-04-12 02:54:24,661 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally acquired 2023-04-12 02:54:24,661 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44679,1681268031772' joining acquired barrier for procedure (testSplitTableRegionDeletedRowsDaughter) in zk 2023-04-12 02:54:24,661 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionDeletedRowsDaughter is still present, removing 2023-04-12 02:54:24,661 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,661 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionDeletedRowsDaughter' with timeout 60000ms 2023-04-12 02:54:24,661 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:54:24,661 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' starting 'acquire' stage 2023-04-12 02:54:24,663 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,663 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,663 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,664 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 02:54:24,664 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,664 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 02:54:24,665 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 02:54:24,665 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 02:54:24,665 DEBUG [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testSplitTableRegionDeletedRowsDaughter,,1681268062677.1c6b8f18205322cbb248ea4fb10155d8. 2023-04-12 02:54:24,665 DEBUG [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testSplitTableRegionDeletedRowsDaughter,,1681268062677.1c6b8f18205322cbb248ea4fb10155d8. started... 2023-04-12 02:54:24,665 INFO [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1c6b8f18205322cbb248ea4fb10155d8 2/2 column families, dataSize=530 B heapSize=2.33 KB 2023-04-12 02:54:24,689 INFO [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=265 B at sequenceid=83 (bloomFilter=true), to=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/.tmp/cf1/fdff24c7084643c5a7298e4db0f23442 2023-04-12 02:54:24,698 INFO [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for fdff24c7084643c5a7298e4db0f23442 2023-04-12 02:54:24,735 INFO [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=265 B at sequenceid=83 (bloomFilter=true), to=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/.tmp/cf2/a01c4e9f61f6405380a578694839ec53 2023-04-12 02:54:24,739 INFO [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for a01c4e9f61f6405380a578694839ec53 2023-04-12 02:54:24,741 DEBUG [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/.tmp/cf1/fdff24c7084643c5a7298e4db0f23442 as hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/cf1/fdff24c7084643c5a7298e4db0f23442 2023-04-12 02:54:24,746 INFO [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for fdff24c7084643c5a7298e4db0f23442 2023-04-12 02:54:24,746 INFO [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/cf1/fdff24c7084643c5a7298e4db0f23442, entries=9, sequenceid=83, filesize=5.1 K 2023-04-12 02:54:24,748 DEBUG [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/.tmp/cf2/a01c4e9f61f6405380a578694839ec53 as hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/cf2/a01c4e9f61f6405380a578694839ec53 2023-04-12 02:54:24,755 INFO [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for a01c4e9f61f6405380a578694839ec53 2023-04-12 02:54:24,755 INFO [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/cf2/a01c4e9f61f6405380a578694839ec53, entries=9, sequenceid=83, filesize=5.1 K 2023-04-12 02:54:24,756 INFO [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~530 B/530, heapSize ~2.30 KB/2352, currentSize=0 B/0 for 1c6b8f18205322cbb248ea4fb10155d8 in 91ms, sequenceid=83, compaction requested=true 2023-04-12 02:54:24,756 DEBUG [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1c6b8f18205322cbb248ea4fb10155d8: 2023-04-12 02:54:24,757 DEBUG [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testSplitTableRegionDeletedRowsDaughter,,1681268062677.1c6b8f18205322cbb248ea4fb10155d8. 2023-04-12 02:54:24,757 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 02:54:24,757 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 02:54:24,757 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:24,757 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally acquired 2023-04-12 02:54:24,757 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34579,1681268031719' joining acquired barrier for procedure (testSplitTableRegionDeletedRowsDaughter) in zk 2023-04-12 02:54:24,766 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:24,767 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:24,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 02:54:24,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 02:54:24,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 02:54:24,767 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,767 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 02:54:24,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 02:54:24,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:24,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:24,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:24,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 02:54:24,771 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34579,1681268031719' joining acquired barrier for procedure 'testSplitTableRegionDeletedRowsDaughter' on coordinator 2023-04-12 02:54:24,771 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@39ac44d9[Count = 0] remaining members to acquire global barrier 2023-04-12 02:54:24,771 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testSplitTableRegionDeletedRowsDaughter' starting 'in-barrier' execution. 2023-04-12 02:54:24,771 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,772 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,772 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,772 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,772 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,773 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:24,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,773 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' received 'reached' from coordinator. 2023-04-12 02:54:24,773 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' received 'reached' from coordinator. 2023-04-12 02:54:24,773 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally completed 2023-04-12 02:54:24,773 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionDeletedRowsDaughter' completed for member 'jenkins-hbase4.apache.org,34579,1681268031719' in zk 2023-04-12 02:54:24,773 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally completed 2023-04-12 02:54:24,773 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 02:54:24,773 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionDeletedRowsDaughter' completed for member 'jenkins-hbase4.apache.org,44679,1681268031772' in zk 2023-04-12 02:54:24,773 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,773 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,773 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' received 'reached' from coordinator. 2023-04-12 02:54:24,773 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally completed 2023-04-12 02:54:24,773 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionDeletedRowsDaughter' completed for member 'jenkins-hbase4.apache.org,43091,1681268031804' in zk 2023-04-12 02:54:24,776 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' has notified controller of completion 2023-04-12 02:54:24,776 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:54:24,776 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' completed. 2023-04-12 02:54:24,776 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:24,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:24,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 02:54:24,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 02:54:24,780 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' has notified controller of completion 2023-04-12 02:54:24,780 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:54:24,780 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' completed. 2023-04-12 02:54:24,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 02:54:24,781 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' has notified controller of completion 2023-04-12 02:54:24,781 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:54:24,781 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' completed. 2023-04-12 02:54:24,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 02:54:24,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:24,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:24,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:24,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 02:54:24,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:24,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:24,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:24,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testSplitTableRegionDeletedRowsDaughter' member 'jenkins-hbase4.apache.org,34579,1681268031719': 2023-04-12 02:54:24,785 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34579,1681268031719' released barrier for procedure'testSplitTableRegionDeletedRowsDaughter', counting down latch. Waiting for 0 more 2023-04-12 02:54:24,785 INFO [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testSplitTableRegionDeletedRowsDaughter' execution completed 2023-04-12 02:54:24,785 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 02:54:24,785 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 02:54:24,785 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,785 INFO [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testSplitTableRegionDeletedRowsDaughterincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 02:54:24,786 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,786 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,786 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:54:24,787 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,787 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:54:24,787 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,787 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,787 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,787 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:54:24,787 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 02:54:24,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 02:54:24,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,787 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:54:24,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:54:24,787 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:54:24,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:54:24,788 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:54:24,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:54:24,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 02:54:24,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,789 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:24,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,789 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:24,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 02:54:24,789 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:24,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:24,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:24,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:24,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 02:54:24,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,791 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:24,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:24,792 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:24,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:24,792 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:24,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:24,797 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:54:24,797 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:54:24,797 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:54:24,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:54:24,797 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:54:24,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:54:24,797 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:54:24,797 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:24,797 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:54:24,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:54:24,797 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,797 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:54:24,798 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:54:24,798 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:54:24,798 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:54:24,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:54:24,798 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:24,798 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:54:24,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:54:24,798 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:24,798 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:54:24,798 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:54:24,798 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testSplitTableRegionDeletedRowsDaughter' 2023-04-12 02:54:24,798 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 02:54:24,798 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,798 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:24,798 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,798 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:24,798 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:24,799 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,799 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:54:24,799 DEBUG [Listener at localhost/46667] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testSplitTableRegionDeletedRowsDaughter'' to complete. (max 20000 ms per retry) 2023-04-12 02:54:24,799 DEBUG [Listener at localhost/46667] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 02:54:24,899 DEBUG [Listener at localhost/46667] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 02:54:24,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 02:54:24,920 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 02:54:24,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] procedure.ProcedureCoordinator(143): Procedure testSplitTableRegionDeletedRowsDaughter was in running list but was completed. Accepting new attempt. 2023-04-12 02:54:24,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] procedure.ProcedureCoordinator(165): Submitting procedure testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,926 INFO [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testSplitTableRegionDeletedRowsDaughter' 2023-04-12 02:54:24,926 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:54:24,926 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testSplitTableRegionDeletedRowsDaughter' starting 'acquire' 2023-04-12 02:54:24,926 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testSplitTableRegionDeletedRowsDaughter', kicking off acquire phase on members. 2023-04-12 02:54:24,926 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,927 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,929 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:54:24,929 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:24,929 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:54:24,929 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:54:24,929 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:54:24,929 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:54:24,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:54:24,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:54:24,929 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:24,929 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 02:54:24,929 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:54:24,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:54:24,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,930 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,930 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,930 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 02:54:24,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,931 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 02:54:24,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 02:54:24,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,931 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,932 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionDeletedRowsDaughter is still present, removing 2023-04-12 02:54:24,931 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionDeletedRowsDaughter is still present, removing 2023-04-12 02:54:24,932 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,932 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionDeletedRowsDaughter' with timeout 60000ms 2023-04-12 02:54:24,932 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,932 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:54:24,932 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' starting 'acquire' stage 2023-04-12 02:54:24,932 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,932 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionDeletedRowsDaughter' with timeout 60000ms 2023-04-12 02:54:24,932 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:54:24,932 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' starting 'acquire' stage 2023-04-12 02:54:24,932 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally acquired 2023-04-12 02:54:24,932 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43091,1681268031804' joining acquired barrier for procedure (testSplitTableRegionDeletedRowsDaughter) in zk 2023-04-12 02:54:24,934 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,934 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,934 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 02:54:24,941 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionDeletedRowsDaughter is still present, removing 2023-04-12 02:54:24,941 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,941 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionDeletedRowsDaughter' with timeout 60000ms 2023-04-12 02:54:24,941 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:54:24,941 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' starting 'acquire' stage 2023-04-12 02:54:24,941 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally acquired 2023-04-12 02:54:24,941 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44679,1681268031772' joining acquired barrier for procedure (testSplitTableRegionDeletedRowsDaughter) in zk 2023-04-12 02:54:24,942 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 02:54:24,942 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 02:54:24,942 DEBUG [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testSplitTableRegionDeletedRowsDaughter,,1681268062677.1c6b8f18205322cbb248ea4fb10155d8. 2023-04-12 02:54:24,942 DEBUG [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testSplitTableRegionDeletedRowsDaughter,,1681268062677.1c6b8f18205322cbb248ea4fb10155d8. started... 2023-04-12 02:54:24,942 INFO [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1c6b8f18205322cbb248ea4fb10155d8 2/2 column families, dataSize=250 B heapSize=1.52 KB 2023-04-12 02:54:24,942 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,943 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:24,943 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 02:54:24,971 INFO [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=125 B at sequenceid=91 (bloomFilter=true), to=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/.tmp/cf1/4868a2af61a94040a8fcc176f05bb0fc 2023-04-12 02:54:24,977 INFO [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 4868a2af61a94040a8fcc176f05bb0fc 2023-04-12 02:54:24,999 INFO [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=125 B at sequenceid=91 (bloomFilter=true), to=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/.tmp/cf2/0c6c01c95a4f4bc3a4312a2eee9e8eec 2023-04-12 02:54:25,004 INFO [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 0c6c01c95a4f4bc3a4312a2eee9e8eec 2023-04-12 02:54:25,006 DEBUG [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/.tmp/cf1/4868a2af61a94040a8fcc176f05bb0fc as hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/cf1/4868a2af61a94040a8fcc176f05bb0fc 2023-04-12 02:54:25,014 INFO [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 4868a2af61a94040a8fcc176f05bb0fc 2023-04-12 02:54:25,014 INFO [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/cf1/4868a2af61a94040a8fcc176f05bb0fc, entries=5, sequenceid=91, filesize=5.0 K 2023-04-12 02:54:25,015 DEBUG [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/.tmp/cf2/0c6c01c95a4f4bc3a4312a2eee9e8eec as hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/cf2/0c6c01c95a4f4bc3a4312a2eee9e8eec 2023-04-12 02:54:25,021 INFO [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 0c6c01c95a4f4bc3a4312a2eee9e8eec 2023-04-12 02:54:25,022 INFO [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/cf2/0c6c01c95a4f4bc3a4312a2eee9e8eec, entries=5, sequenceid=91, filesize=5.0 K 2023-04-12 02:54:25,023 INFO [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~250 B/250, heapSize ~1.48 KB/1520, currentSize=0 B/0 for 1c6b8f18205322cbb248ea4fb10155d8 in 81ms, sequenceid=91, compaction requested=true 2023-04-12 02:54:25,023 DEBUG [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1c6b8f18205322cbb248ea4fb10155d8: 2023-04-12 02:54:25,023 DEBUG [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testSplitTableRegionDeletedRowsDaughter,,1681268062677.1c6b8f18205322cbb248ea4fb10155d8. 2023-04-12 02:54:25,023 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 02:54:25,023 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 02:54:25,023 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:25,023 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally acquired 2023-04-12 02:54:25,023 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34579,1681268031719' joining acquired barrier for procedure (testSplitTableRegionDeletedRowsDaughter) in zk 2023-04-12 02:54:25,025 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:25,025 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:25,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:25,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 02:54:25,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 02:54:25,025 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:25,025 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 02:54:25,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 02:54:25,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 02:54:25,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:25,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:25,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:25,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:25,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 02:54:25,027 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34579,1681268031719' joining acquired barrier for procedure 'testSplitTableRegionDeletedRowsDaughter' on coordinator 2023-04-12 02:54:25,027 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@35451ddd[Count = 0] remaining members to acquire global barrier 2023-04-12 02:54:25,027 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testSplitTableRegionDeletedRowsDaughter' starting 'in-barrier' execution. 2023-04-12 02:54:25,027 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:25,028 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:25,028 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:25,028 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:25,028 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:25,029 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:25,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:25,028 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:25,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:25,029 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' received 'reached' from coordinator. 2023-04-12 02:54:25,029 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally completed 2023-04-12 02:54:25,029 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' received 'reached' from coordinator. 2023-04-12 02:54:25,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:25,029 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:25,029 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 02:54:25,029 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally completed 2023-04-12 02:54:25,029 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionDeletedRowsDaughter' completed for member 'jenkins-hbase4.apache.org,34579,1681268031719' in zk 2023-04-12 02:54:25,029 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionDeletedRowsDaughter' completed for member 'jenkins-hbase4.apache.org,44679,1681268031772' in zk 2023-04-12 02:54:25,029 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' received 'reached' from coordinator. 2023-04-12 02:54:25,029 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally completed 2023-04-12 02:54:25,029 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionDeletedRowsDaughter' completed for member 'jenkins-hbase4.apache.org,43091,1681268031804' in zk 2023-04-12 02:54:25,031 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' has notified controller of completion 2023-04-12 02:54:25,031 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:54:25,031 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' completed. 2023-04-12 02:54:25,033 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:25,033 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' has notified controller of completion 2023-04-12 02:54:25,033 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:54:25,033 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' has notified controller of completion 2023-04-12 02:54:25,033 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' completed. 2023-04-12 02:54:25,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:25,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 02:54:25,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 02:54:25,033 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:54:25,033 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' completed. 2023-04-12 02:54:25,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 02:54:25,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 02:54:25,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:25,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:25,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:25,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:25,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 02:54:25,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:25,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:25,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:25,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:25,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testSplitTableRegionDeletedRowsDaughter' member 'jenkins-hbase4.apache.org,34579,1681268031719': 2023-04-12 02:54:25,039 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34579,1681268031719' released barrier for procedure'testSplitTableRegionDeletedRowsDaughter', counting down latch. Waiting for 0 more 2023-04-12 02:54:25,039 INFO [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testSplitTableRegionDeletedRowsDaughter' execution completed 2023-04-12 02:54:25,039 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 02:54:25,039 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 02:54:25,039 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:25,039 INFO [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testSplitTableRegionDeletedRowsDaughterincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 02:54:25,040 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:25,040 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:54:25,040 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:25,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:25,040 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:25,041 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:25,041 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:54:25,041 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:25,041 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:25,041 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:54:25,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:54:25,041 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:25,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:25,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:25,041 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:54:25,041 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:25,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:25,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:25,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 02:54:25,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 02:54:25,042 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:54:25,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:54:25,042 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:54:25,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:54:25,042 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:25,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:25,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 02:54:25,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:25,042 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:25,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:25,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 02:54:25,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:25,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:25,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:25,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:25,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 02:54:25,044 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:25,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:25,045 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:25,045 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:25,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:25,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:25,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:25,051 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:54:25,051 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:54:25,051 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:54:25,051 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:54:25,051 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:54:25,052 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:54:25,052 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:54:25,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:54:25,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:54:25,052 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:54:25,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:54:25,052 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:54:25,052 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:54:25,052 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:54:25,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:54:25,052 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:54:25,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:54:25,052 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testSplitTableRegionDeletedRowsDaughter' 2023-04-12 02:54:25,053 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 02:54:25,052 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:25,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:54:25,053 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:25,053 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:25,053 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:25,053 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:25,053 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:25,053 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:25,053 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:25,053 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:25,053 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:25,053 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:25,052 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:54:25,053 DEBUG [Listener at localhost/46667] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testSplitTableRegionDeletedRowsDaughter'' to complete. (max 20000 ms per retry) 2023-04-12 02:54:25,053 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter because node does not exist (not an error) 2023-04-12 02:54:25,053 DEBUG [Listener at localhost/46667] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 02:54:25,154 DEBUG [Listener at localhost/46667] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 02:54:25,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 02:54:25,176 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 02:54:25,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] procedure.ProcedureCoordinator(143): Procedure testSplitTableRegionDeletedRowsDaughter was in running list but was completed. Accepting new attempt. 2023-04-12 02:54:25,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] procedure.ProcedureCoordinator(165): Submitting procedure testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:25,188 INFO [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testSplitTableRegionDeletedRowsDaughter' 2023-04-12 02:54:25,188 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:54:25,188 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testSplitTableRegionDeletedRowsDaughter' starting 'acquire' 2023-04-12 02:54:25,188 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testSplitTableRegionDeletedRowsDaughter', kicking off acquire phase on members. 2023-04-12 02:54:25,189 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:25,189 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:25,191 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:54:25,191 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:54:25,191 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:54:25,191 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:54:25,191 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:25,191 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:54:25,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:54:25,191 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:54:25,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:54:25,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:54:25,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:25,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:25,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:25,192 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:25,192 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:25,192 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:25,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 02:54:25,193 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:25,193 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 02:54:25,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 02:54:25,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:25,193 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:25,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 02:54:25,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:25,194 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:25,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:25,194 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:25,194 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionDeletedRowsDaughter is still present, removing 2023-04-12 02:54:25,194 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:25,194 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionDeletedRowsDaughter' with timeout 60000ms 2023-04-12 02:54:25,194 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:54:25,194 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' starting 'acquire' stage 2023-04-12 02:54:25,195 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionDeletedRowsDaughter is still present, removing 2023-04-12 02:54:25,196 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:25,196 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionDeletedRowsDaughter' with timeout 60000ms 2023-04-12 02:54:25,196 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:54:25,196 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' starting 'acquire' stage 2023-04-12 02:54:25,196 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally acquired 2023-04-12 02:54:25,196 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44679,1681268031772' joining acquired barrier for procedure (testSplitTableRegionDeletedRowsDaughter) in zk 2023-04-12 02:54:25,197 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:25,198 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:25,198 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 02:54:25,204 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionDeletedRowsDaughter is still present, removing 2023-04-12 02:54:25,205 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:25,205 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionDeletedRowsDaughter' with timeout 60000ms 2023-04-12 02:54:25,205 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:54:25,205 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' starting 'acquire' stage 2023-04-12 02:54:25,205 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally acquired 2023-04-12 02:54:25,205 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43091,1681268031804' joining acquired barrier for procedure (testSplitTableRegionDeletedRowsDaughter) in zk 2023-04-12 02:54:25,206 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:25,207 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:25,207 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 02:54:25,208 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 02:54:25,208 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 02:54:25,208 DEBUG [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testSplitTableRegionDeletedRowsDaughter,,1681268062677.1c6b8f18205322cbb248ea4fb10155d8. 2023-04-12 02:54:25,208 DEBUG [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testSplitTableRegionDeletedRowsDaughter,,1681268062677.1c6b8f18205322cbb248ea4fb10155d8. started... 2023-04-12 02:54:25,208 INFO [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1c6b8f18205322cbb248ea4fb10155d8 2/2 column families, dataSize=250 B heapSize=1.52 KB 2023-04-12 02:54:25,253 INFO [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=125 B at sequenceid=99 (bloomFilter=true), to=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/.tmp/cf1/85930b8d05664956906c5fa532cccb71 2023-04-12 02:54:25,260 INFO [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 85930b8d05664956906c5fa532cccb71 2023-04-12 02:54:25,284 INFO [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=125 B at sequenceid=99 (bloomFilter=true), to=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/.tmp/cf2/37083e44dea546c8a8710fe5148a6dcd 2023-04-12 02:54:25,290 INFO [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 37083e44dea546c8a8710fe5148a6dcd 2023-04-12 02:54:25,291 DEBUG [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/.tmp/cf1/85930b8d05664956906c5fa532cccb71 as hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/cf1/85930b8d05664956906c5fa532cccb71 2023-04-12 02:54:25,296 INFO [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 85930b8d05664956906c5fa532cccb71 2023-04-12 02:54:25,296 INFO [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/cf1/85930b8d05664956906c5fa532cccb71, entries=5, sequenceid=99, filesize=5.0 K 2023-04-12 02:54:25,297 DEBUG [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/.tmp/cf2/37083e44dea546c8a8710fe5148a6dcd as hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/cf2/37083e44dea546c8a8710fe5148a6dcd 2023-04-12 02:54:25,303 INFO [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 37083e44dea546c8a8710fe5148a6dcd 2023-04-12 02:54:25,303 INFO [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/cf2/37083e44dea546c8a8710fe5148a6dcd, entries=5, sequenceid=99, filesize=5.0 K 2023-04-12 02:54:25,304 INFO [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~250 B/250, heapSize ~1.48 KB/1520, currentSize=0 B/0 for 1c6b8f18205322cbb248ea4fb10155d8 in 96ms, sequenceid=99, compaction requested=true 2023-04-12 02:54:25,304 DEBUG [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1c6b8f18205322cbb248ea4fb10155d8: 2023-04-12 02:54:25,304 DEBUG [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testSplitTableRegionDeletedRowsDaughter,,1681268062677.1c6b8f18205322cbb248ea4fb10155d8. 2023-04-12 02:54:25,304 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 02:54:25,304 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 02:54:25,304 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:25,304 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally acquired 2023-04-12 02:54:25,305 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34579,1681268031719' joining acquired barrier for procedure (testSplitTableRegionDeletedRowsDaughter) in zk 2023-04-12 02:54:25,306 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:25,306 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:25,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:25,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 02:54:25,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 02:54:25,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 02:54:25,307 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:25,307 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 02:54:25,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 02:54:25,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:25,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:25,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:25,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:25,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 02:54:25,309 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34579,1681268031719' joining acquired barrier for procedure 'testSplitTableRegionDeletedRowsDaughter' on coordinator 2023-04-12 02:54:25,309 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@134321bf[Count = 0] remaining members to acquire global barrier 2023-04-12 02:54:25,309 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testSplitTableRegionDeletedRowsDaughter' starting 'in-barrier' execution. 2023-04-12 02:54:25,309 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:25,311 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:25,311 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:25,312 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:25,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:25,312 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:25,312 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' received 'reached' from coordinator. 2023-04-12 02:54:25,311 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:25,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:25,312 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally completed 2023-04-12 02:54:25,312 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:25,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:25,312 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionDeletedRowsDaughter' completed for member 'jenkins-hbase4.apache.org,43091,1681268031804' in zk 2023-04-12 02:54:25,312 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' received 'reached' from coordinator. 2023-04-12 02:54:25,312 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally completed 2023-04-12 02:54:25,312 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionDeletedRowsDaughter' completed for member 'jenkins-hbase4.apache.org,34579,1681268031719' in zk 2023-04-12 02:54:25,312 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' received 'reached' from coordinator. 2023-04-12 02:54:25,312 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally completed 2023-04-12 02:54:25,312 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionDeletedRowsDaughter' completed for member 'jenkins-hbase4.apache.org,44679,1681268031772' in zk 2023-04-12 02:54:25,313 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:25,313 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 02:54:25,314 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' has notified controller of completion 2023-04-12 02:54:25,314 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:54:25,314 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' completed. 2023-04-12 02:54:25,314 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' has notified controller of completion 2023-04-12 02:54:25,314 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:54:25,314 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' completed. 2023-04-12 02:54:25,314 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:25,314 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' has notified controller of completion 2023-04-12 02:54:25,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:25,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 02:54:25,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 02:54:25,314 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:54:25,315 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' completed. 2023-04-12 02:54:25,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 02:54:25,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 02:54:25,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:25,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:25,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:25,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:25,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 02:54:25,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:25,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:25,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:25,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:25,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testSplitTableRegionDeletedRowsDaughter' member 'jenkins-hbase4.apache.org,34579,1681268031719': 2023-04-12 02:54:25,321 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34579,1681268031719' released barrier for procedure'testSplitTableRegionDeletedRowsDaughter', counting down latch. Waiting for 0 more 2023-04-12 02:54:25,321 INFO [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testSplitTableRegionDeletedRowsDaughter' execution completed 2023-04-12 02:54:25,321 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 02:54:25,321 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 02:54:25,321 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:25,321 INFO [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testSplitTableRegionDeletedRowsDaughterincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 02:54:25,324 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:25,324 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:54:25,324 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:25,324 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:25,324 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:25,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:25,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:25,324 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:54:25,325 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:25,325 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:54:25,325 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:25,325 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:25,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:25,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:25,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 02:54:25,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 02:54:25,325 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:54:25,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:54:25,325 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:54:25,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:54:25,326 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:54:25,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:25,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:54:25,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 02:54:25,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:25,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:25,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:25,327 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:25,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 02:54:25,327 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:25,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:25,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:25,330 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:25,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:25,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:25,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 02:54:25,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:25,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:25,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:25,332 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:25,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:25,333 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:25,334 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:25,337 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:25,337 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:25,337 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:25,337 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:25,337 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:54:25,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:54:25,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:54:25,337 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:54:25,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:54:25,337 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:54:25,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:54:25,337 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:54:25,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:54:25,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:54:25,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:54:25,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:54:25,338 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:54:25,339 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:25,339 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:54:25,339 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:54:25,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:54:25,339 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:25,339 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:25,339 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:54:25,339 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:25,339 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:54:25,339 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:25,339 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:25,339 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testSplitTableRegionDeletedRowsDaughter' 2023-04-12 02:54:25,339 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 02:54:25,339 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:54:25,339 DEBUG [Listener at localhost/46667] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testSplitTableRegionDeletedRowsDaughter'' to complete. (max 20000 ms per retry) 2023-04-12 02:54:25,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:54:25,340 DEBUG [Listener at localhost/46667] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 02:54:25,440 DEBUG [Listener at localhost/46667] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 02:54:25,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 02:54:25,445 DEBUG [Listener at localhost/46667] assignment.SplitTableRegionProcedure(219): Splittable=true state=OPEN, location=jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:25,446 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(1029): Stored pid=72, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testSplitTableRegionDeletedRowsDaughter, parent=1c6b8f18205322cbb248ea4fb10155d8, daughterA=81296e35c0d27ffa1ba7676ff53f87e1, daughterB=72b78dfd0678a920ddc3700612ff3932 2023-04-12 02:54:25,448 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=72, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testSplitTableRegionDeletedRowsDaughter, parent=1c6b8f18205322cbb248ea4fb10155d8, daughterA=81296e35c0d27ffa1ba7676ff53f87e1, daughterB=72b78dfd0678a920ddc3700612ff3932 2023-04-12 02:54:25,448 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=72, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testSplitTableRegionDeletedRowsDaughter, parent=1c6b8f18205322cbb248ea4fb10155d8, daughterA=81296e35c0d27ffa1ba7676ff53f87e1, daughterB=72b78dfd0678a920ddc3700612ff3932 2023-04-12 02:54:25,448 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=72, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testSplitTableRegionDeletedRowsDaughter, parent=1c6b8f18205322cbb248ea4fb10155d8, daughterA=81296e35c0d27ffa1ba7676ff53f87e1, daughterB=72b78dfd0678a920ddc3700612ff3932 2023-04-12 02:54:25,450 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=73, ppid=72, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitTableRegionDeletedRowsDaughter, region=1c6b8f18205322cbb248ea4fb10155d8, UNASSIGN}] 2023-04-12 02:54:25,450 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=73, ppid=72, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitTableRegionDeletedRowsDaughter, region=1c6b8f18205322cbb248ea4fb10155d8, UNASSIGN 2023-04-12 02:54:25,451 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=73 updating hbase:meta row=1c6b8f18205322cbb248ea4fb10155d8, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:25,451 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitTableRegionDeletedRowsDaughter,,1681268062677.1c6b8f18205322cbb248ea4fb10155d8.","families":{"info":[{"qualifier":"regioninfo","vlen":73,"tag":[],"timestamp":"1681268065451"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681268065451"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681268065451"}]},"ts":"1681268065451"} 2023-04-12 02:54:25,453 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=74, ppid=73, state=RUNNABLE; CloseRegionProcedure 1c6b8f18205322cbb248ea4fb10155d8, server=jenkins-hbase4.apache.org,34579,1681268031719}] 2023-04-12 02:54:25,606 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 1c6b8f18205322cbb248ea4fb10155d8 2023-04-12 02:54:25,606 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 1c6b8f18205322cbb248ea4fb10155d8, disabling compactions & flushes 2023-04-12 02:54:25,606 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region testSplitTableRegionDeletedRowsDaughter,,1681268062677.1c6b8f18205322cbb248ea4fb10155d8. 2023-04-12 02:54:25,606 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testSplitTableRegionDeletedRowsDaughter,,1681268062677.1c6b8f18205322cbb248ea4fb10155d8. 2023-04-12 02:54:25,606 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on testSplitTableRegionDeletedRowsDaughter,,1681268062677.1c6b8f18205322cbb248ea4fb10155d8. after waiting 0 ms 2023-04-12 02:54:25,606 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region testSplitTableRegionDeletedRowsDaughter,,1681268062677.1c6b8f18205322cbb248ea4fb10155d8. 2023-04-12 02:54:25,606 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 1c6b8f18205322cbb248ea4fb10155d8 2/2 column families, dataSize=50 B heapSize=720 B 2023-04-12 02:54:25,647 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=25 B at sequenceid=103 (bloomFilter=true), to=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/.tmp/cf1/45831fa527a845c783019394c67c0fa7 2023-04-12 02:54:25,652 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 45831fa527a845c783019394c67c0fa7 2023-04-12 02:54:25,695 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=25 B at sequenceid=103 (bloomFilter=true), to=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/.tmp/cf2/8ac99bc8ac8f4c3b873212471489db78 2023-04-12 02:54:25,700 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 8ac99bc8ac8f4c3b873212471489db78 2023-04-12 02:54:25,701 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/.tmp/cf1/45831fa527a845c783019394c67c0fa7 as hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/cf1/45831fa527a845c783019394c67c0fa7 2023-04-12 02:54:25,712 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 45831fa527a845c783019394c67c0fa7 2023-04-12 02:54:25,712 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/cf1/45831fa527a845c783019394c67c0fa7, entries=1, sequenceid=103, filesize=4.9 K 2023-04-12 02:54:25,713 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/.tmp/cf2/8ac99bc8ac8f4c3b873212471489db78 as hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/cf2/8ac99bc8ac8f4c3b873212471489db78 2023-04-12 02:54:25,717 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 8ac99bc8ac8f4c3b873212471489db78 2023-04-12 02:54:25,718 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/cf2/8ac99bc8ac8f4c3b873212471489db78, entries=1, sequenceid=103, filesize=4.9 K 2023-04-12 02:54:25,718 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~50 B/50, heapSize ~688 B/688, currentSize=0 B/0 for 1c6b8f18205322cbb248ea4fb10155d8 in 112ms, sequenceid=103, compaction requested=true 2023-04-12 02:54:25,734 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/recovered.edits/106.seqid, newMaxSeqId=106, maxSeqId=1 2023-04-12 02:54:25,735 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro 2023-04-12 02:54:25,735 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed testSplitTableRegionDeletedRowsDaughter,,1681268062677.1c6b8f18205322cbb248ea4fb10155d8. 2023-04-12 02:54:25,735 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 1c6b8f18205322cbb248ea4fb10155d8: 2023-04-12 02:54:25,737 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 1c6b8f18205322cbb248ea4fb10155d8 2023-04-12 02:54:25,737 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=73 updating hbase:meta row=1c6b8f18205322cbb248ea4fb10155d8, regionState=CLOSED 2023-04-12 02:54:25,737 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testSplitTableRegionDeletedRowsDaughter,,1681268062677.1c6b8f18205322cbb248ea4fb10155d8.","families":{"info":[{"qualifier":"regioninfo","vlen":73,"tag":[],"timestamp":"1681268065737"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681268065737"}]},"ts":"1681268065737"} 2023-04-12 02:54:25,740 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=74, resume processing ppid=73 2023-04-12 02:54:25,740 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=74, ppid=73, state=SUCCESS; CloseRegionProcedure 1c6b8f18205322cbb248ea4fb10155d8, server=jenkins-hbase4.apache.org,34579,1681268031719 in 286 msec 2023-04-12 02:54:25,743 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=73, resume processing ppid=72 2023-04-12 02:54:25,743 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=73, ppid=72, state=SUCCESS; TransitRegionStateProcedure table=testSplitTableRegionDeletedRowsDaughter, region=1c6b8f18205322cbb248ea4fb10155d8, UNASSIGN in 291 msec 2023-04-12 02:54:25,761 INFO [PEWorker-1] assignment.SplitTableRegionProcedure(694): pid=72 splitting 20 storefiles, region=1c6b8f18205322cbb248ea4fb10155d8, threads=16 2023-04-12 02:54:25,770 DEBUG [StoreFileSplitter-pool-0] assignment.SplitTableRegionProcedure(776): pid=72 splitting started for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/cf2/0c6c01c95a4f4bc3a4312a2eee9e8eec for region: 1c6b8f18205322cbb248ea4fb10155d8 2023-04-12 02:54:25,772 DEBUG [StoreFileSplitter-pool-1] assignment.SplitTableRegionProcedure(776): pid=72 splitting started for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/cf2/37083e44dea546c8a8710fe5148a6dcd for region: 1c6b8f18205322cbb248ea4fb10155d8 2023-04-12 02:54:25,776 DEBUG [StoreFileSplitter-pool-2] assignment.SplitTableRegionProcedure(776): pid=72 splitting started for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/cf2/381b6116e3b64f31a2b235e9f7067d52 for region: 1c6b8f18205322cbb248ea4fb10155d8 2023-04-12 02:54:25,779 INFO [StoreFileSplitter-pool-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 0c6c01c95a4f4bc3a4312a2eee9e8eec 2023-04-12 02:54:25,780 DEBUG [StoreFileSplitter-pool-3] assignment.SplitTableRegionProcedure(776): pid=72 splitting started for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/cf2/77b0532593204cbd90904f2bbec317e7 for region: 1c6b8f18205322cbb248ea4fb10155d8 2023-04-12 02:54:25,783 DEBUG [StoreFileSplitter-pool-4] assignment.SplitTableRegionProcedure(776): pid=72 splitting started for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/cf2/85cb745e6ba6488f8367339705326ef9 for region: 1c6b8f18205322cbb248ea4fb10155d8 2023-04-12 02:54:25,783 DEBUG [StoreFileSplitter-pool-5] assignment.SplitTableRegionProcedure(776): pid=72 splitting started for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/cf2/8ac99bc8ac8f4c3b873212471489db78 for region: 1c6b8f18205322cbb248ea4fb10155d8 2023-04-12 02:54:25,784 DEBUG [StoreFileSplitter-pool-7] assignment.SplitTableRegionProcedure(776): pid=72 splitting started for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/cf2/a01c4e9f61f6405380a578694839ec53 for region: 1c6b8f18205322cbb248ea4fb10155d8 2023-04-12 02:54:25,786 DEBUG [StoreFileSplitter-pool-9] assignment.SplitTableRegionProcedure(776): pid=72 splitting started for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/cf2/ff9f5d4721ca4fc5a30cfe3141f45ca7 for region: 1c6b8f18205322cbb248ea4fb10155d8 2023-04-12 02:54:25,786 DEBUG [StoreFileSplitter-pool-6] assignment.SplitTableRegionProcedure(776): pid=72 splitting started for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/cf2/926ca93402364365a7ac53b2e60b9bf5 for region: 1c6b8f18205322cbb248ea4fb10155d8 2023-04-12 02:54:25,787 DEBUG [StoreFileSplitter-pool-8] assignment.SplitTableRegionProcedure(776): pid=72 splitting started for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/cf2/e6cbb96ab13a4b67a21da11e8efd233f for region: 1c6b8f18205322cbb248ea4fb10155d8 2023-04-12 02:54:25,788 DEBUG [StoreFileSplitter-pool-10] assignment.SplitTableRegionProcedure(776): pid=72 splitting started for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/cf1/0fd593a3724149b8ad466c2cef9ba846 for region: 1c6b8f18205322cbb248ea4fb10155d8 2023-04-12 02:54:25,792 DEBUG [StoreFileSplitter-pool-11] assignment.SplitTableRegionProcedure(776): pid=72 splitting started for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/cf1/45831fa527a845c783019394c67c0fa7 for region: 1c6b8f18205322cbb248ea4fb10155d8 2023-04-12 02:54:25,800 DEBUG [StoreFileSplitter-pool-12] assignment.SplitTableRegionProcedure(776): pid=72 splitting started for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/cf1/4868a2af61a94040a8fcc176f05bb0fc for region: 1c6b8f18205322cbb248ea4fb10155d8 2023-04-12 02:54:25,800 DEBUG [StoreFileSplitter-pool-13] assignment.SplitTableRegionProcedure(776): pid=72 splitting started for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/cf1/710424cef2c64d2ebc740c81294a95cb for region: 1c6b8f18205322cbb248ea4fb10155d8 2023-04-12 02:54:25,801 DEBUG [StoreFileSplitter-pool-14] assignment.SplitTableRegionProcedure(776): pid=72 splitting started for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/cf1/789e1a943f304f83be48e491529ca4ad for region: 1c6b8f18205322cbb248ea4fb10155d8 2023-04-12 02:54:25,804 DEBUG [StoreFileSplitter-pool-15] assignment.SplitTableRegionProcedure(776): pid=72 splitting started for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/cf1/85930b8d05664956906c5fa532cccb71 for region: 1c6b8f18205322cbb248ea4fb10155d8 2023-04-12 02:54:25,812 DEBUG [StoreFileSplitter-pool-4] regionserver.HRegionFileSystem(713): Will create HFileLink file for hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/cf2/85cb745e6ba6488f8367339705326ef9, top=false 2023-04-12 02:54:25,813 INFO [StoreFileSplitter-pool-1] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 37083e44dea546c8a8710fe5148a6dcd 2023-04-12 02:54:25,813 INFO [StoreFileSplitter-pool-7] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for a01c4e9f61f6405380a578694839ec53 2023-04-12 02:54:25,814 DEBUG [StoreFileSplitter-pool-9] regionserver.HRegionFileSystem(713): Will create HFileLink file for hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/cf2/ff9f5d4721ca4fc5a30cfe3141f45ca7, top=false 2023-04-12 02:54:25,814 INFO [StoreFileSplitter-pool-5] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 8ac99bc8ac8f4c3b873212471489db78 2023-04-12 02:54:25,815 INFO [StoreFileSplitter-pool-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 0c6c01c95a4f4bc3a4312a2eee9e8eec 2023-04-12 02:54:25,815 DEBUG [StoreFileSplitter-pool-0] regionserver.HRegionFileSystem(700): Will create HFileLink file for hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/cf2/0c6c01c95a4f4bc3a4312a2eee9e8eec, top=true 2023-04-12 02:54:25,817 DEBUG [StoreFileSplitter-pool-8] regionserver.HRegionFileSystem(713): Will create HFileLink file for hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/cf2/e6cbb96ab13a4b67a21da11e8efd233f, top=false 2023-04-12 02:54:25,825 DEBUG [StoreFileSplitter-pool-13] regionserver.HRegionFileSystem(713): Will create HFileLink file for hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/cf1/710424cef2c64d2ebc740c81294a95cb, top=false 2023-04-12 02:54:25,832 INFO [StoreFileSplitter-pool-1] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 37083e44dea546c8a8710fe5148a6dcd 2023-04-12 02:54:25,832 DEBUG [StoreFileSplitter-pool-1] regionserver.HRegionFileSystem(700): Will create HFileLink file for hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/cf2/37083e44dea546c8a8710fe5148a6dcd, top=true 2023-04-12 02:54:25,832 INFO [StoreFileSplitter-pool-11] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 45831fa527a845c783019394c67c0fa7 2023-04-12 02:54:25,834 INFO [StoreFileSplitter-pool-15] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 85930b8d05664956906c5fa532cccb71 2023-04-12 02:54:25,838 INFO [StoreFileSplitter-pool-5] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 8ac99bc8ac8f4c3b873212471489db78 2023-04-12 02:54:25,838 DEBUG [StoreFileSplitter-pool-5] regionserver.HRegionFileSystem(700): Will create HFileLink file for hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/cf2/8ac99bc8ac8f4c3b873212471489db78, top=true 2023-04-12 02:54:25,839 INFO [StoreFileSplitter-pool-12] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 4868a2af61a94040a8fcc176f05bb0fc 2023-04-12 02:54:25,843 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-12 02:54:25,910 INFO [StoreFileSplitter-pool-11] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 45831fa527a845c783019394c67c0fa7 2023-04-12 02:54:25,910 DEBUG [StoreFileSplitter-pool-11] regionserver.HRegionFileSystem(700): Will create HFileLink file for hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/cf1/45831fa527a845c783019394c67c0fa7, top=true 2023-04-12 02:54:25,910 INFO [StoreFileSplitter-pool-15] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 85930b8d05664956906c5fa532cccb71 2023-04-12 02:54:25,910 DEBUG [StoreFileSplitter-pool-15] regionserver.HRegionFileSystem(700): Will create HFileLink file for hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/cf1/85930b8d05664956906c5fa532cccb71, top=true 2023-04-12 02:54:25,913 INFO [StoreFileSplitter-pool-12] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 4868a2af61a94040a8fcc176f05bb0fc 2023-04-12 02:54:25,913 DEBUG [StoreFileSplitter-pool-12] regionserver.HRegionFileSystem(700): Will create HFileLink file for hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/cf1/4868a2af61a94040a8fcc176f05bb0fc, top=true 2023-04-12 02:54:26,052 INFO [StoreFileSplitter-pool-4] regionserver.HRegionFileSystem(742): Created linkFile:hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/.splits/81296e35c0d27ffa1ba7676ff53f87e1/cf2/testSplitTableRegionDeletedRowsDaughter=1c6b8f18205322cbb248ea4fb10155d8-85cb745e6ba6488f8367339705326ef9 for child: 81296e35c0d27ffa1ba7676ff53f87e1, parent: 1c6b8f18205322cbb248ea4fb10155d8 2023-04-12 02:54:26,052 INFO [StoreFileSplitter-pool-9] regionserver.HRegionFileSystem(742): Created linkFile:hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/.splits/81296e35c0d27ffa1ba7676ff53f87e1/cf2/testSplitTableRegionDeletedRowsDaughter=1c6b8f18205322cbb248ea4fb10155d8-ff9f5d4721ca4fc5a30cfe3141f45ca7 for child: 81296e35c0d27ffa1ba7676ff53f87e1, parent: 1c6b8f18205322cbb248ea4fb10155d8 2023-04-12 02:54:26,052 INFO [StoreFileSplitter-pool-8] regionserver.HRegionFileSystem(742): Created linkFile:hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/.splits/81296e35c0d27ffa1ba7676ff53f87e1/cf2/testSplitTableRegionDeletedRowsDaughter=1c6b8f18205322cbb248ea4fb10155d8-e6cbb96ab13a4b67a21da11e8efd233f for child: 81296e35c0d27ffa1ba7676ff53f87e1, parent: 1c6b8f18205322cbb248ea4fb10155d8 2023-04-12 02:54:26,052 INFO [StoreFileSplitter-pool-13] regionserver.HRegionFileSystem(742): Created linkFile:hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/.splits/81296e35c0d27ffa1ba7676ff53f87e1/cf1/testSplitTableRegionDeletedRowsDaughter=1c6b8f18205322cbb248ea4fb10155d8-710424cef2c64d2ebc740c81294a95cb for child: 81296e35c0d27ffa1ba7676ff53f87e1, parent: 1c6b8f18205322cbb248ea4fb10155d8 2023-04-12 02:54:26,052 INFO [StoreFileSplitter-pool-0] regionserver.HRegionFileSystem(742): Created linkFile:hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/.splits/72b78dfd0678a920ddc3700612ff3932/cf2/testSplitTableRegionDeletedRowsDaughter=1c6b8f18205322cbb248ea4fb10155d8-0c6c01c95a4f4bc3a4312a2eee9e8eec for child: 72b78dfd0678a920ddc3700612ff3932, parent: 1c6b8f18205322cbb248ea4fb10155d8 2023-04-12 02:54:26,053 DEBUG [StoreFileSplitter-pool-0] assignment.SplitTableRegionProcedure(787): pid=72 splitting complete for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/cf2/0c6c01c95a4f4bc3a4312a2eee9e8eec for region: 1c6b8f18205322cbb248ea4fb10155d8 2023-04-12 02:54:26,053 DEBUG [StoreFileSplitter-pool-0] assignment.SplitTableRegionProcedure(776): pid=72 splitting started for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/cf1/985388b30d784536a1f270feb7f52366 for region: 1c6b8f18205322cbb248ea4fb10155d8 2023-04-12 02:54:26,053 INFO [StoreFileSplitter-pool-15] regionserver.HRegionFileSystem(742): Created linkFile:hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/.splits/72b78dfd0678a920ddc3700612ff3932/cf1/testSplitTableRegionDeletedRowsDaughter=1c6b8f18205322cbb248ea4fb10155d8-85930b8d05664956906c5fa532cccb71 for child: 72b78dfd0678a920ddc3700612ff3932, parent: 1c6b8f18205322cbb248ea4fb10155d8 2023-04-12 02:54:26,053 DEBUG [StoreFileSplitter-pool-15] assignment.SplitTableRegionProcedure(787): pid=72 splitting complete for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/cf1/85930b8d05664956906c5fa532cccb71 for region: 1c6b8f18205322cbb248ea4fb10155d8 2023-04-12 02:54:26,053 DEBUG [StoreFileSplitter-pool-15] assignment.SplitTableRegionProcedure(776): pid=72 splitting started for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/cf1/9c4efc02782d420291d1fe480cade0d3 for region: 1c6b8f18205322cbb248ea4fb10155d8 2023-04-12 02:54:26,053 INFO [StoreFileSplitter-pool-12] regionserver.HRegionFileSystem(742): Created linkFile:hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/.splits/72b78dfd0678a920ddc3700612ff3932/cf1/testSplitTableRegionDeletedRowsDaughter=1c6b8f18205322cbb248ea4fb10155d8-4868a2af61a94040a8fcc176f05bb0fc for child: 72b78dfd0678a920ddc3700612ff3932, parent: 1c6b8f18205322cbb248ea4fb10155d8 2023-04-12 02:54:26,053 DEBUG [StoreFileSplitter-pool-12] assignment.SplitTableRegionProcedure(787): pid=72 splitting complete for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/cf1/4868a2af61a94040a8fcc176f05bb0fc for region: 1c6b8f18205322cbb248ea4fb10155d8 2023-04-12 02:54:26,053 DEBUG [StoreFileSplitter-pool-12] assignment.SplitTableRegionProcedure(776): pid=72 splitting started for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/cf1/bce8056f97d34999b5a16bbacddae4f5 for region: 1c6b8f18205322cbb248ea4fb10155d8 2023-04-12 02:54:26,053 INFO [StoreFileSplitter-pool-11] regionserver.HRegionFileSystem(742): Created linkFile:hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/.splits/72b78dfd0678a920ddc3700612ff3932/cf1/testSplitTableRegionDeletedRowsDaughter=1c6b8f18205322cbb248ea4fb10155d8-45831fa527a845c783019394c67c0fa7 for child: 72b78dfd0678a920ddc3700612ff3932, parent: 1c6b8f18205322cbb248ea4fb10155d8 2023-04-12 02:54:26,053 DEBUG [StoreFileSplitter-pool-11] assignment.SplitTableRegionProcedure(787): pid=72 splitting complete for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/cf1/45831fa527a845c783019394c67c0fa7 for region: 1c6b8f18205322cbb248ea4fb10155d8 2023-04-12 02:54:26,053 DEBUG [StoreFileSplitter-pool-11] assignment.SplitTableRegionProcedure(776): pid=72 splitting started for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/cf1/fdff24c7084643c5a7298e4db0f23442 for region: 1c6b8f18205322cbb248ea4fb10155d8 2023-04-12 02:54:26,054 INFO [StoreFileSplitter-pool-5] regionserver.HRegionFileSystem(742): Created linkFile:hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/.splits/72b78dfd0678a920ddc3700612ff3932/cf2/testSplitTableRegionDeletedRowsDaughter=1c6b8f18205322cbb248ea4fb10155d8-8ac99bc8ac8f4c3b873212471489db78 for child: 72b78dfd0678a920ddc3700612ff3932, parent: 1c6b8f18205322cbb248ea4fb10155d8 2023-04-12 02:54:26,054 DEBUG [StoreFileSplitter-pool-5] assignment.SplitTableRegionProcedure(787): pid=72 splitting complete for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/cf2/8ac99bc8ac8f4c3b873212471489db78 for region: 1c6b8f18205322cbb248ea4fb10155d8 2023-04-12 02:54:26,081 DEBUG [StoreFileSplitter-pool-8] assignment.SplitTableRegionProcedure(787): pid=72 splitting complete for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/cf2/e6cbb96ab13a4b67a21da11e8efd233f for region: 1c6b8f18205322cbb248ea4fb10155d8 2023-04-12 02:54:26,082 DEBUG [StoreFileSplitter-pool-4] assignment.SplitTableRegionProcedure(787): pid=72 splitting complete for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/cf2/85cb745e6ba6488f8367339705326ef9 for region: 1c6b8f18205322cbb248ea4fb10155d8 2023-04-12 02:54:26,082 DEBUG [StoreFileSplitter-pool-15] regionserver.HRegionFileSystem(713): Will create HFileLink file for hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/cf1/9c4efc02782d420291d1fe480cade0d3, top=false 2023-04-12 02:54:26,083 DEBUG [StoreFileSplitter-pool-13] assignment.SplitTableRegionProcedure(787): pid=72 splitting complete for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/cf1/710424cef2c64d2ebc740c81294a95cb for region: 1c6b8f18205322cbb248ea4fb10155d8 2023-04-12 02:54:26,083 INFO [StoreFileSplitter-pool-11] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for fdff24c7084643c5a7298e4db0f23442 2023-04-12 02:54:26,122 INFO [StoreFileSplitter-pool-1] regionserver.HRegionFileSystem(742): Created linkFile:hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/.splits/72b78dfd0678a920ddc3700612ff3932/cf2/testSplitTableRegionDeletedRowsDaughter=1c6b8f18205322cbb248ea4fb10155d8-37083e44dea546c8a8710fe5148a6dcd for child: 72b78dfd0678a920ddc3700612ff3932, parent: 1c6b8f18205322cbb248ea4fb10155d8 2023-04-12 02:54:26,122 DEBUG [StoreFileSplitter-pool-1] assignment.SplitTableRegionProcedure(787): pid=72 splitting complete for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/cf2/37083e44dea546c8a8710fe5148a6dcd for region: 1c6b8f18205322cbb248ea4fb10155d8 2023-04-12 02:54:26,153 INFO [StoreFileSplitter-pool-15] regionserver.HRegionFileSystem(742): Created linkFile:hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/.splits/81296e35c0d27ffa1ba7676ff53f87e1/cf1/testSplitTableRegionDeletedRowsDaughter=1c6b8f18205322cbb248ea4fb10155d8-9c4efc02782d420291d1fe480cade0d3 for child: 81296e35c0d27ffa1ba7676ff53f87e1, parent: 1c6b8f18205322cbb248ea4fb10155d8 2023-04-12 02:54:26,282 INFO [StoreFileSplitter-pool-7] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for a01c4e9f61f6405380a578694839ec53 2023-04-12 02:54:26,282 DEBUG [StoreFileSplitter-pool-0] regionserver.HRegionFileSystem(713): Will create HFileLink file for hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/cf1/985388b30d784536a1f270feb7f52366, top=false 2023-04-12 02:54:26,282 DEBUG [StoreFileSplitter-pool-9] assignment.SplitTableRegionProcedure(787): pid=72 splitting complete for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/cf2/ff9f5d4721ca4fc5a30cfe3141f45ca7 for region: 1c6b8f18205322cbb248ea4fb10155d8 2023-04-12 02:54:26,285 DEBUG [StoreFileSplitter-pool-15] assignment.SplitTableRegionProcedure(787): pid=72 splitting complete for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/cf1/9c4efc02782d420291d1fe480cade0d3 for region: 1c6b8f18205322cbb248ea4fb10155d8 2023-04-12 02:54:26,294 INFO [StoreFileSplitter-pool-11] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for fdff24c7084643c5a7298e4db0f23442 2023-04-12 02:54:26,364 DEBUG [StoreFileSplitter-pool-3] assignment.SplitTableRegionProcedure(787): pid=72 splitting complete for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/cf2/77b0532593204cbd90904f2bbec317e7 for region: 1c6b8f18205322cbb248ea4fb10155d8 2023-04-12 02:54:26,378 DEBUG [StoreFileSplitter-pool-10] assignment.SplitTableRegionProcedure(787): pid=72 splitting complete for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/cf1/0fd593a3724149b8ad466c2cef9ba846 for region: 1c6b8f18205322cbb248ea4fb10155d8 2023-04-12 02:54:26,393 INFO [StoreFileSplitter-pool-0] regionserver.HRegionFileSystem(742): Created linkFile:hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/.splits/81296e35c0d27ffa1ba7676ff53f87e1/cf1/testSplitTableRegionDeletedRowsDaughter=1c6b8f18205322cbb248ea4fb10155d8-985388b30d784536a1f270feb7f52366 for child: 81296e35c0d27ffa1ba7676ff53f87e1, parent: 1c6b8f18205322cbb248ea4fb10155d8 2023-04-12 02:54:26,398 DEBUG [StoreFileSplitter-pool-0] assignment.SplitTableRegionProcedure(787): pid=72 splitting complete for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/cf1/985388b30d784536a1f270feb7f52366 for region: 1c6b8f18205322cbb248ea4fb10155d8 2023-04-12 02:54:26,475 DEBUG [StoreFileSplitter-pool-2] assignment.SplitTableRegionProcedure(787): pid=72 splitting complete for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/cf2/381b6116e3b64f31a2b235e9f7067d52 for region: 1c6b8f18205322cbb248ea4fb10155d8 2023-04-12 02:54:26,514 DEBUG [StoreFileSplitter-pool-7] assignment.SplitTableRegionProcedure(787): pid=72 splitting complete for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/cf2/a01c4e9f61f6405380a578694839ec53 for region: 1c6b8f18205322cbb248ea4fb10155d8 2023-04-12 02:54:26,519 DEBUG [StoreFileSplitter-pool-6] assignment.SplitTableRegionProcedure(787): pid=72 splitting complete for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/cf2/926ca93402364365a7ac53b2e60b9bf5 for region: 1c6b8f18205322cbb248ea4fb10155d8 2023-04-12 02:54:26,520 DEBUG [StoreFileSplitter-pool-14] assignment.SplitTableRegionProcedure(787): pid=72 splitting complete for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/cf1/789e1a943f304f83be48e491529ca4ad for region: 1c6b8f18205322cbb248ea4fb10155d8 2023-04-12 02:54:26,546 DEBUG [StoreFileSplitter-pool-12] assignment.SplitTableRegionProcedure(787): pid=72 splitting complete for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/cf1/bce8056f97d34999b5a16bbacddae4f5 for region: 1c6b8f18205322cbb248ea4fb10155d8 2023-04-12 02:54:26,936 DEBUG [StoreFileSplitter-pool-11] assignment.SplitTableRegionProcedure(787): pid=72 splitting complete for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/cf1/fdff24c7084643c5a7298e4db0f23442 for region: 1c6b8f18205322cbb248ea4fb10155d8 2023-04-12 02:54:26,936 DEBUG [PEWorker-1] assignment.SplitTableRegionProcedure(755): pid=72 split storefiles for region 1c6b8f18205322cbb248ea4fb10155d8 Daughter A: 14 storefiles, Daughter B: 14 storefiles. 2023-04-12 02:54:27,034 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/81296e35c0d27ffa1ba7676ff53f87e1/recovered.edits/106.seqid, newMaxSeqId=106, maxSeqId=-1 2023-04-12 02:54:27,037 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/72b78dfd0678a920ddc3700612ff3932/recovered.edits/106.seqid, newMaxSeqId=106, maxSeqId=-1 2023-04-12 02:54:27,040 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":3,"row":"testSplitTableRegionDeletedRowsDaughter,,1681268062677.1c6b8f18205322cbb248ea4fb10155d8.","families":{"info":[{"qualifier":"regioninfo","vlen":73,"tag":[],"timestamp":"1681268067039"},{"qualifier":"splitA","vlen":75,"tag":[],"timestamp":"1681268067039"},{"qualifier":"splitB","vlen":75,"tag":[],"timestamp":"1681268067039"}]},"ts":"1681268067039"} 2023-04-12 02:54:27,040 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":3,"row":"testSplitTableRegionDeletedRowsDaughter,,1681268065445.81296e35c0d27ffa1ba7676ff53f87e1.","families":{"info":[{"qualifier":"regioninfo","vlen":75,"tag":[],"timestamp":"1681268067039"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681268067039"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681268067039"}]},"ts":"1681268067039"} 2023-04-12 02:54:27,040 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":3,"row":"testSplitTableRegionDeletedRowsDaughter,60,1681268065445.72b78dfd0678a920ddc3700612ff3932.","families":{"info":[{"qualifier":"regioninfo","vlen":75,"tag":[],"timestamp":"1681268067039"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681268067039"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681268067039"}]},"ts":"1681268067039"} 2023-04-12 02:54:27,047 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=75, ppid=72, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSplitTableRegionDeletedRowsDaughter, region=81296e35c0d27ffa1ba7676ff53f87e1, ASSIGN}, {pid=76, ppid=72, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSplitTableRegionDeletedRowsDaughter, region=72b78dfd0678a920ddc3700612ff3932, ASSIGN}] 2023-04-12 02:54:27,048 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=76, ppid=72, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSplitTableRegionDeletedRowsDaughter, region=72b78dfd0678a920ddc3700612ff3932, ASSIGN 2023-04-12 02:54:27,049 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=76, ppid=72, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testSplitTableRegionDeletedRowsDaughter, region=72b78dfd0678a920ddc3700612ff3932, ASSIGN; state=SPLITTING_NEW, location=jenkins-hbase4.apache.org,34579,1681268031719; forceNewPlan=false, retain=false 2023-04-12 02:54:27,050 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=75, ppid=72, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSplitTableRegionDeletedRowsDaughter, region=81296e35c0d27ffa1ba7676ff53f87e1, ASSIGN 2023-04-12 02:54:27,050 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=75, ppid=72, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testSplitTableRegionDeletedRowsDaughter, region=81296e35c0d27ffa1ba7676ff53f87e1, ASSIGN; state=SPLITTING_NEW, location=jenkins-hbase4.apache.org,34579,1681268031719; forceNewPlan=false, retain=false 2023-04-12 02:54:27,199 INFO [jenkins-hbase4:40281] balancer.BaseLoadBalancer(1545): Reassigned 2 regions. 2 retained the pre-restart assignment. 2023-04-12 02:54:27,201 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=75 updating hbase:meta row=81296e35c0d27ffa1ba7676ff53f87e1, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:27,201 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitTableRegionDeletedRowsDaughter,,1681268065445.81296e35c0d27ffa1ba7676ff53f87e1.","families":{"info":[{"qualifier":"regioninfo","vlen":75,"tag":[],"timestamp":"1681268067201"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681268067201"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681268067201"}]},"ts":"1681268067201"} 2023-04-12 02:54:27,203 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=77, ppid=75, state=RUNNABLE; OpenRegionProcedure 81296e35c0d27ffa1ba7676ff53f87e1, server=jenkins-hbase4.apache.org,34579,1681268031719}] 2023-04-12 02:54:27,207 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=76 updating hbase:meta row=72b78dfd0678a920ddc3700612ff3932, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:27,207 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitTableRegionDeletedRowsDaughter,60,1681268065445.72b78dfd0678a920ddc3700612ff3932.","families":{"info":[{"qualifier":"regioninfo","vlen":75,"tag":[],"timestamp":"1681268067207"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681268067207"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681268067207"}]},"ts":"1681268067207"} 2023-04-12 02:54:27,209 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=78, ppid=76, state=RUNNABLE; OpenRegionProcedure 72b78dfd0678a920ddc3700612ff3932, server=jenkins-hbase4.apache.org,34579,1681268031719}] 2023-04-12 02:54:27,359 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testSplitTableRegionDeletedRowsDaughter,,1681268065445.81296e35c0d27ffa1ba7676ff53f87e1. 2023-04-12 02:54:27,359 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 81296e35c0d27ffa1ba7676ff53f87e1, NAME => 'testSplitTableRegionDeletedRowsDaughter,,1681268065445.81296e35c0d27ffa1ba7676ff53f87e1.', STARTKEY => '', ENDKEY => '60'} 2023-04-12 02:54:27,359 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(174): System coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro loaded, priority=536870911. 2023-04-12 02:54:27,359 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testSplitTableRegionDeletedRowsDaughter 81296e35c0d27ffa1ba7676ff53f87e1 2023-04-12 02:54:27,359 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated testSplitTableRegionDeletedRowsDaughter,,1681268065445.81296e35c0d27ffa1ba7676ff53f87e1.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-12 02:54:27,359 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 81296e35c0d27ffa1ba7676ff53f87e1 2023-04-12 02:54:27,359 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 81296e35c0d27ffa1ba7676ff53f87e1 2023-04-12 02:54:27,361 INFO [StoreOpener-81296e35c0d27ffa1ba7676ff53f87e1-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf1 of region 81296e35c0d27ffa1ba7676ff53f87e1 2023-04-12 02:54:27,362 DEBUG [StoreOpener-81296e35c0d27ffa1ba7676ff53f87e1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/81296e35c0d27ffa1ba7676ff53f87e1/cf1 2023-04-12 02:54:27,362 DEBUG [StoreOpener-81296e35c0d27ffa1ba7676ff53f87e1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/81296e35c0d27ffa1ba7676ff53f87e1/cf1 2023-04-12 02:54:27,362 INFO [StoreOpener-81296e35c0d27ffa1ba7676ff53f87e1-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 81296e35c0d27ffa1ba7676ff53f87e1 columnFamilyName cf1 2023-04-12 02:54:27,374 DEBUG [StoreOpener-81296e35c0d27ffa1ba7676ff53f87e1-1] regionserver.HStore(539): loaded hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/81296e35c0d27ffa1ba7676ff53f87e1/cf1/0fd593a3724149b8ad466c2cef9ba846.1c6b8f18205322cbb248ea4fb10155d8->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/cf1/0fd593a3724149b8ad466c2cef9ba846-bottom 2023-04-12 02:54:27,379 DEBUG [StoreOpener-81296e35c0d27ffa1ba7676ff53f87e1-1] regionserver.HStore(539): loaded hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/81296e35c0d27ffa1ba7676ff53f87e1/cf1/789e1a943f304f83be48e491529ca4ad.1c6b8f18205322cbb248ea4fb10155d8->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/cf1/789e1a943f304f83be48e491529ca4ad-bottom 2023-04-12 02:54:27,383 DEBUG [StoreOpener-81296e35c0d27ffa1ba7676ff53f87e1-1] regionserver.HStore(539): loaded hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/81296e35c0d27ffa1ba7676ff53f87e1/cf1/bce8056f97d34999b5a16bbacddae4f5.1c6b8f18205322cbb248ea4fb10155d8->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/cf1/bce8056f97d34999b5a16bbacddae4f5-bottom 2023-04-12 02:54:27,386 INFO [StoreFileOpener-cf1-1] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for fdff24c7084643c5a7298e4db0f23442.1c6b8f18205322cbb248ea4fb10155d8 2023-04-12 02:54:27,388 DEBUG [StoreOpener-81296e35c0d27ffa1ba7676ff53f87e1-1] regionserver.HStore(539): loaded hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/81296e35c0d27ffa1ba7676ff53f87e1/cf1/fdff24c7084643c5a7298e4db0f23442.1c6b8f18205322cbb248ea4fb10155d8->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/cf1/fdff24c7084643c5a7298e4db0f23442-bottom 2023-04-12 02:54:27,391 DEBUG [StoreOpener-81296e35c0d27ffa1ba7676ff53f87e1-1] regionserver.HStore(539): loaded hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/81296e35c0d27ffa1ba7676ff53f87e1/cf1/testSplitTableRegionDeletedRowsDaughter=1c6b8f18205322cbb248ea4fb10155d8-710424cef2c64d2ebc740c81294a95cb 2023-04-12 02:54:27,395 DEBUG [StoreOpener-81296e35c0d27ffa1ba7676ff53f87e1-1] regionserver.HStore(539): loaded hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/81296e35c0d27ffa1ba7676ff53f87e1/cf1/testSplitTableRegionDeletedRowsDaughter=1c6b8f18205322cbb248ea4fb10155d8-985388b30d784536a1f270feb7f52366 2023-04-12 02:54:27,402 DEBUG [StoreOpener-81296e35c0d27ffa1ba7676ff53f87e1-1] regionserver.HStore(539): loaded hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/81296e35c0d27ffa1ba7676ff53f87e1/cf1/testSplitTableRegionDeletedRowsDaughter=1c6b8f18205322cbb248ea4fb10155d8-9c4efc02782d420291d1fe480cade0d3 2023-04-12 02:54:27,402 INFO [StoreOpener-81296e35c0d27ffa1ba7676ff53f87e1-1] regionserver.HStore(310): Store=81296e35c0d27ffa1ba7676ff53f87e1/cf1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-12 02:54:27,403 INFO [StoreOpener-81296e35c0d27ffa1ba7676ff53f87e1-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf2 of region 81296e35c0d27ffa1ba7676ff53f87e1 2023-04-12 02:54:27,404 DEBUG [StoreOpener-81296e35c0d27ffa1ba7676ff53f87e1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/81296e35c0d27ffa1ba7676ff53f87e1/cf2 2023-04-12 02:54:27,404 DEBUG [StoreOpener-81296e35c0d27ffa1ba7676ff53f87e1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/81296e35c0d27ffa1ba7676ff53f87e1/cf2 2023-04-12 02:54:27,405 INFO [StoreOpener-81296e35c0d27ffa1ba7676ff53f87e1-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 81296e35c0d27ffa1ba7676ff53f87e1 columnFamilyName cf2 2023-04-12 02:54:27,417 DEBUG [StoreOpener-81296e35c0d27ffa1ba7676ff53f87e1-1] regionserver.HStore(539): loaded hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/81296e35c0d27ffa1ba7676ff53f87e1/cf2/381b6116e3b64f31a2b235e9f7067d52.1c6b8f18205322cbb248ea4fb10155d8->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/cf2/381b6116e3b64f31a2b235e9f7067d52-bottom 2023-04-12 02:54:27,422 DEBUG [StoreOpener-81296e35c0d27ffa1ba7676ff53f87e1-1] regionserver.HStore(539): loaded hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/81296e35c0d27ffa1ba7676ff53f87e1/cf2/77b0532593204cbd90904f2bbec317e7.1c6b8f18205322cbb248ea4fb10155d8->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/cf2/77b0532593204cbd90904f2bbec317e7-bottom 2023-04-12 02:54:27,426 DEBUG [StoreOpener-81296e35c0d27ffa1ba7676ff53f87e1-1] regionserver.HStore(539): loaded hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/81296e35c0d27ffa1ba7676ff53f87e1/cf2/926ca93402364365a7ac53b2e60b9bf5.1c6b8f18205322cbb248ea4fb10155d8->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/cf2/926ca93402364365a7ac53b2e60b9bf5-bottom 2023-04-12 02:54:27,430 INFO [StoreFileOpener-cf2-1] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for a01c4e9f61f6405380a578694839ec53.1c6b8f18205322cbb248ea4fb10155d8 2023-04-12 02:54:27,431 DEBUG [StoreOpener-81296e35c0d27ffa1ba7676ff53f87e1-1] regionserver.HStore(539): loaded hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/81296e35c0d27ffa1ba7676ff53f87e1/cf2/a01c4e9f61f6405380a578694839ec53.1c6b8f18205322cbb248ea4fb10155d8->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/cf2/a01c4e9f61f6405380a578694839ec53-bottom 2023-04-12 02:54:27,435 DEBUG [StoreOpener-81296e35c0d27ffa1ba7676ff53f87e1-1] regionserver.HStore(539): loaded hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/81296e35c0d27ffa1ba7676ff53f87e1/cf2/testSplitTableRegionDeletedRowsDaughter=1c6b8f18205322cbb248ea4fb10155d8-85cb745e6ba6488f8367339705326ef9 2023-04-12 02:54:27,438 DEBUG [StoreOpener-81296e35c0d27ffa1ba7676ff53f87e1-1] regionserver.HStore(539): loaded hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/81296e35c0d27ffa1ba7676ff53f87e1/cf2/testSplitTableRegionDeletedRowsDaughter=1c6b8f18205322cbb248ea4fb10155d8-e6cbb96ab13a4b67a21da11e8efd233f 2023-04-12 02:54:27,445 DEBUG [StoreOpener-81296e35c0d27ffa1ba7676ff53f87e1-1] regionserver.HStore(539): loaded hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/81296e35c0d27ffa1ba7676ff53f87e1/cf2/testSplitTableRegionDeletedRowsDaughter=1c6b8f18205322cbb248ea4fb10155d8-ff9f5d4721ca4fc5a30cfe3141f45ca7 2023-04-12 02:54:27,445 INFO [StoreOpener-81296e35c0d27ffa1ba7676ff53f87e1-1] regionserver.HStore(310): Store=81296e35c0d27ffa1ba7676ff53f87e1/cf2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-12 02:54:27,446 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/81296e35c0d27ffa1ba7676ff53f87e1 2023-04-12 02:54:27,447 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/81296e35c0d27ffa1ba7676ff53f87e1 2023-04-12 02:54:27,450 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testSplitTableRegionDeletedRowsDaughter descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-04-12 02:54:27,451 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 81296e35c0d27ffa1ba7676ff53f87e1 2023-04-12 02:54:27,452 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 81296e35c0d27ffa1ba7676ff53f87e1; next sequenceid=107; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=66202294, jitterRate=-0.01350894570350647}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-04-12 02:54:27,452 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 81296e35c0d27ffa1ba7676ff53f87e1: 2023-04-12 02:54:27,453 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testSplitTableRegionDeletedRowsDaughter,,1681268065445.81296e35c0d27ffa1ba7676ff53f87e1., pid=77, masterSystemTime=1681268067355 2023-04-12 02:54:27,453 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: Opening Region; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 02:54:27,455 DEBUG [RS:0;jenkins-hbase4:34579-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-12 02:54:27,455 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: Opening Region; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 02:54:27,455 INFO [RS:0;jenkins-hbase4:34579-shortCompactions-0] regionserver.HStore(1898): Keeping/Overriding Compaction request priority to -2147482648 for CF cf1 since it belongs to recently split daughter region testSplitTableRegionDeletedRowsDaughter,,1681268065445.81296e35c0d27ffa1ba7676ff53f87e1. 2023-04-12 02:54:27,455 DEBUG [RS:0;jenkins-hbase4:34579-shortCompactions-0] regionserver.HStore(1912): 81296e35c0d27ffa1ba7676ff53f87e1/cf1 is initiating minor compaction (all files) 2023-04-12 02:54:27,455 DEBUG [RS:0;jenkins-hbase4:34579-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-12 02:54:27,455 INFO [RS:0;jenkins-hbase4:34579-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 81296e35c0d27ffa1ba7676ff53f87e1/cf1 in testSplitTableRegionDeletedRowsDaughter,,1681268065445.81296e35c0d27ffa1ba7676ff53f87e1. 2023-04-12 02:54:27,455 INFO [RS:0;jenkins-hbase4:34579-longCompactions-0] regionserver.HStore(1898): Keeping/Overriding Compaction request priority to -2147482648 for CF cf2 since it belongs to recently split daughter region testSplitTableRegionDeletedRowsDaughter,,1681268065445.81296e35c0d27ffa1ba7676ff53f87e1. 2023-04-12 02:54:27,455 DEBUG [RS:0;jenkins-hbase4:34579-longCompactions-0] regionserver.HStore(1912): 81296e35c0d27ffa1ba7676ff53f87e1/cf2 is initiating minor compaction (all files) 2023-04-12 02:54:27,455 INFO [RS:0;jenkins-hbase4:34579-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 81296e35c0d27ffa1ba7676ff53f87e1/cf2 in testSplitTableRegionDeletedRowsDaughter,,1681268065445.81296e35c0d27ffa1ba7676ff53f87e1. 2023-04-12 02:54:27,455 INFO [RS:0;jenkins-hbase4:34579-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/81296e35c0d27ffa1ba7676ff53f87e1/cf1/789e1a943f304f83be48e491529ca4ad.1c6b8f18205322cbb248ea4fb10155d8->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/cf1/789e1a943f304f83be48e491529ca4ad-bottom, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/81296e35c0d27ffa1ba7676ff53f87e1/cf1/0fd593a3724149b8ad466c2cef9ba846.1c6b8f18205322cbb248ea4fb10155d8->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/cf1/0fd593a3724149b8ad466c2cef9ba846-bottom, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/81296e35c0d27ffa1ba7676ff53f87e1/cf1/bce8056f97d34999b5a16bbacddae4f5.1c6b8f18205322cbb248ea4fb10155d8->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/cf1/bce8056f97d34999b5a16bbacddae4f5-bottom, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/81296e35c0d27ffa1ba7676ff53f87e1/cf1/testSplitTableRegionDeletedRowsDaughter=1c6b8f18205322cbb248ea4fb10155d8-985388b30d784536a1f270feb7f52366, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/81296e35c0d27ffa1ba7676ff53f87e1/cf1/testSplitTableRegionDeletedRowsDaughter=1c6b8f18205322cbb248ea4fb10155d8-710424cef2c64d2ebc740c81294a95cb, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/81296e35c0d27ffa1ba7676ff53f87e1/cf1/testSplitTableRegionDeletedRowsDaughter=1c6b8f18205322cbb248ea4fb10155d8-9c4efc02782d420291d1fe480cade0d3, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/81296e35c0d27ffa1ba7676ff53f87e1/cf1/fdff24c7084643c5a7298e4db0f23442.1c6b8f18205322cbb248ea4fb10155d8->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/cf1/fdff24c7084643c5a7298e4db0f23442-bottom] into tmpdir=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/81296e35c0d27ffa1ba7676ff53f87e1/.tmp, totalSize=35.0 K 2023-04-12 02:54:27,455 INFO [RS:0;jenkins-hbase4:34579-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/81296e35c0d27ffa1ba7676ff53f87e1/cf2/926ca93402364365a7ac53b2e60b9bf5.1c6b8f18205322cbb248ea4fb10155d8->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/cf2/926ca93402364365a7ac53b2e60b9bf5-bottom, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/81296e35c0d27ffa1ba7676ff53f87e1/cf2/77b0532593204cbd90904f2bbec317e7.1c6b8f18205322cbb248ea4fb10155d8->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/cf2/77b0532593204cbd90904f2bbec317e7-bottom, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/81296e35c0d27ffa1ba7676ff53f87e1/cf2/381b6116e3b64f31a2b235e9f7067d52.1c6b8f18205322cbb248ea4fb10155d8->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/cf2/381b6116e3b64f31a2b235e9f7067d52-bottom, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/81296e35c0d27ffa1ba7676ff53f87e1/cf2/testSplitTableRegionDeletedRowsDaughter=1c6b8f18205322cbb248ea4fb10155d8-ff9f5d4721ca4fc5a30cfe3141f45ca7, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/81296e35c0d27ffa1ba7676ff53f87e1/cf2/testSplitTableRegionDeletedRowsDaughter=1c6b8f18205322cbb248ea4fb10155d8-85cb745e6ba6488f8367339705326ef9, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/81296e35c0d27ffa1ba7676ff53f87e1/cf2/testSplitTableRegionDeletedRowsDaughter=1c6b8f18205322cbb248ea4fb10155d8-e6cbb96ab13a4b67a21da11e8efd233f, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/81296e35c0d27ffa1ba7676ff53f87e1/cf2/a01c4e9f61f6405380a578694839ec53.1c6b8f18205322cbb248ea4fb10155d8->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/cf2/a01c4e9f61f6405380a578694839ec53-bottom] into tmpdir=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/81296e35c0d27ffa1ba7676ff53f87e1/.tmp, totalSize=35.0 K 2023-04-12 02:54:27,456 DEBUG [RS:0;jenkins-hbase4:34579-shortCompactions-0] compactions.Compactor(207): Compacting 789e1a943f304f83be48e491529ca4ad.1c6b8f18205322cbb248ea4fb10155d8, keycount=1, bloomtype=ROW, size=4.8 K, encoding=NONE, compression=NONE, seqNum=6, earliestPutTs=1681268063181 2023-04-12 02:54:27,456 DEBUG [RS:0;jenkins-hbase4:34579-longCompactions-0] compactions.Compactor(207): Compacting 926ca93402364365a7ac53b2e60b9bf5.1c6b8f18205322cbb248ea4fb10155d8, keycount=1, bloomtype=ROW, size=4.8 K, encoding=NONE, compression=NONE, seqNum=6, earliestPutTs=1681268063181 2023-04-12 02:54:27,457 DEBUG [RS:0;jenkins-hbase4:34579-shortCompactions-0] compactions.Compactor(207): Compacting 0fd593a3724149b8ad466c2cef9ba846.1c6b8f18205322cbb248ea4fb10155d8, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=19, earliestPutTs=1681268063370 2023-04-12 02:54:27,457 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testSplitTableRegionDeletedRowsDaughter,,1681268065445.81296e35c0d27ffa1ba7676ff53f87e1. 2023-04-12 02:54:27,457 DEBUG [RS:0;jenkins-hbase4:34579-longCompactions-0] compactions.Compactor(207): Compacting 77b0532593204cbd90904f2bbec317e7.1c6b8f18205322cbb248ea4fb10155d8, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=19, earliestPutTs=1681268063370 2023-04-12 02:54:27,457 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testSplitTableRegionDeletedRowsDaughter,,1681268065445.81296e35c0d27ffa1ba7676ff53f87e1. 2023-04-12 02:54:27,458 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testSplitTableRegionDeletedRowsDaughter,60,1681268065445.72b78dfd0678a920ddc3700612ff3932. 2023-04-12 02:54:27,458 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=75 updating hbase:meta row=81296e35c0d27ffa1ba7676ff53f87e1, regionState=OPEN, openSeqNum=107, regionLocation=jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:27,458 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 72b78dfd0678a920ddc3700612ff3932, NAME => 'testSplitTableRegionDeletedRowsDaughter,60,1681268065445.72b78dfd0678a920ddc3700612ff3932.', STARTKEY => '60', ENDKEY => ''} 2023-04-12 02:54:27,457 DEBUG [RS:0;jenkins-hbase4:34579-shortCompactions-0] compactions.Compactor(207): Compacting bce8056f97d34999b5a16bbacddae4f5.1c6b8f18205322cbb248ea4fb10155d8, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=32, earliestPutTs=1681268063570 2023-04-12 02:54:27,458 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testSplitTableRegionDeletedRowsDaughter,,1681268065445.81296e35c0d27ffa1ba7676ff53f87e1.","families":{"info":[{"qualifier":"regioninfo","vlen":75,"tag":[],"timestamp":"1681268067458"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1681268067458"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681268067458"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681268067458"}]},"ts":"1681268067458"} 2023-04-12 02:54:27,458 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(174): System coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro loaded, priority=536870911. 2023-04-12 02:54:27,459 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testSplitTableRegionDeletedRowsDaughter 72b78dfd0678a920ddc3700612ff3932 2023-04-12 02:54:27,459 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated testSplitTableRegionDeletedRowsDaughter,60,1681268065445.72b78dfd0678a920ddc3700612ff3932.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-12 02:54:27,459 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 72b78dfd0678a920ddc3700612ff3932 2023-04-12 02:54:27,459 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 72b78dfd0678a920ddc3700612ff3932 2023-04-12 02:54:27,460 DEBUG [RS:0;jenkins-hbase4:34579-longCompactions-0] compactions.Compactor(207): Compacting 381b6116e3b64f31a2b235e9f7067d52.1c6b8f18205322cbb248ea4fb10155d8, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=32, earliestPutTs=1681268063570 2023-04-12 02:54:27,460 DEBUG [RS:0;jenkins-hbase4:34579-longCompactions-0] compactions.Compactor(207): Compacting testSplitTableRegionDeletedRowsDaughter=1c6b8f18205322cbb248ea4fb10155d8-ff9f5d4721ca4fc5a30cfe3141f45ca7, keycount=10, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=45, earliestPutTs=1681268063766 2023-04-12 02:54:27,461 DEBUG [RS:0;jenkins-hbase4:34579-shortCompactions-0] compactions.Compactor(207): Compacting testSplitTableRegionDeletedRowsDaughter=1c6b8f18205322cbb248ea4fb10155d8-985388b30d784536a1f270feb7f52366, keycount=10, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=45, earliestPutTs=1681268063766 2023-04-12 02:54:27,461 DEBUG [RS:0;jenkins-hbase4:34579-longCompactions-0] compactions.Compactor(207): Compacting testSplitTableRegionDeletedRowsDaughter=1c6b8f18205322cbb248ea4fb10155d8-85cb745e6ba6488f8367339705326ef9, keycount=10, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=58, earliestPutTs=1681268063958 2023-04-12 02:54:27,461 DEBUG [RS:0;jenkins-hbase4:34579-shortCompactions-0] compactions.Compactor(207): Compacting testSplitTableRegionDeletedRowsDaughter=1c6b8f18205322cbb248ea4fb10155d8-710424cef2c64d2ebc740c81294a95cb, keycount=10, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=58, earliestPutTs=1681268063958 2023-04-12 02:54:27,461 DEBUG [RS:0;jenkins-hbase4:34579-longCompactions-0] compactions.Compactor(207): Compacting testSplitTableRegionDeletedRowsDaughter=1c6b8f18205322cbb248ea4fb10155d8-e6cbb96ab13a4b67a21da11e8efd233f, keycount=10, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=71, earliestPutTs=1681268064291 2023-04-12 02:54:27,462 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=77, resume processing ppid=75 2023-04-12 02:54:27,462 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=77, ppid=75, state=SUCCESS; OpenRegionProcedure 81296e35c0d27ffa1ba7676ff53f87e1, server=jenkins-hbase4.apache.org,34579,1681268031719 in 256 msec 2023-04-12 02:54:27,462 DEBUG [RS:0;jenkins-hbase4:34579-longCompactions-0] compactions.Compactor(207): Compacting a01c4e9f61f6405380a578694839ec53.1c6b8f18205322cbb248ea4fb10155d8, keycount=4, bloomtype=ROW, size=5.1 K, encoding=NONE, compression=NONE, seqNum=83, earliestPutTs=1681268064634 2023-04-12 02:54:27,463 DEBUG [RS:0;jenkins-hbase4:34579-shortCompactions-0] compactions.Compactor(207): Compacting testSplitTableRegionDeletedRowsDaughter=1c6b8f18205322cbb248ea4fb10155d8-9c4efc02782d420291d1fe480cade0d3, keycount=10, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=71, earliestPutTs=1681268064291 2023-04-12 02:54:27,463 DEBUG [RS:0;jenkins-hbase4:34579-shortCompactions-0] compactions.Compactor(207): Compacting fdff24c7084643c5a7298e4db0f23442.1c6b8f18205322cbb248ea4fb10155d8, keycount=4, bloomtype=ROW, size=5.1 K, encoding=NONE, compression=NONE, seqNum=83, earliestPutTs=1681268064634 2023-04-12 02:54:27,464 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=75, ppid=72, state=SUCCESS; TransitRegionStateProcedure table=testSplitTableRegionDeletedRowsDaughter, region=81296e35c0d27ffa1ba7676ff53f87e1, ASSIGN in 415 msec 2023-04-12 02:54:27,464 INFO [StoreOpener-72b78dfd0678a920ddc3700612ff3932-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf1 of region 72b78dfd0678a920ddc3700612ff3932 2023-04-12 02:54:27,466 DEBUG [StoreOpener-72b78dfd0678a920ddc3700612ff3932-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/72b78dfd0678a920ddc3700612ff3932/cf1 2023-04-12 02:54:27,466 DEBUG [StoreOpener-72b78dfd0678a920ddc3700612ff3932-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/72b78dfd0678a920ddc3700612ff3932/cf1 2023-04-12 02:54:27,467 INFO [StoreOpener-72b78dfd0678a920ddc3700612ff3932-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 72b78dfd0678a920ddc3700612ff3932 columnFamilyName cf1 2023-04-12 02:54:27,484 INFO [RS:0;jenkins-hbase4:34579-shortCompactions-0] throttle.PressureAwareThroughputController(145): 81296e35c0d27ffa1ba7676ff53f87e1#cf1#compaction#72 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 02:54:27,489 INFO [RS:0;jenkins-hbase4:34579-longCompactions-0] throttle.PressureAwareThroughputController(145): 81296e35c0d27ffa1ba7676ff53f87e1#cf2#compaction#73 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 02:54:27,490 DEBUG [StoreOpener-72b78dfd0678a920ddc3700612ff3932-1] regionserver.HStore(539): loaded hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/72b78dfd0678a920ddc3700612ff3932/cf1/0fd593a3724149b8ad466c2cef9ba846.1c6b8f18205322cbb248ea4fb10155d8->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/cf1/0fd593a3724149b8ad466c2cef9ba846-top 2023-04-12 02:54:27,498 DEBUG [StoreOpener-72b78dfd0678a920ddc3700612ff3932-1] regionserver.HStore(539): loaded hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/72b78dfd0678a920ddc3700612ff3932/cf1/789e1a943f304f83be48e491529ca4ad.1c6b8f18205322cbb248ea4fb10155d8->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/cf1/789e1a943f304f83be48e491529ca4ad-top 2023-04-12 02:54:27,512 DEBUG [RS:0;jenkins-hbase4:34579-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/81296e35c0d27ffa1ba7676ff53f87e1/.tmp/cf1/072e4ee0f62f45e8ada734db958eabf1 as hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/81296e35c0d27ffa1ba7676ff53f87e1/cf1/072e4ee0f62f45e8ada734db958eabf1 2023-04-12 02:54:27,515 DEBUG [StoreOpener-72b78dfd0678a920ddc3700612ff3932-1] regionserver.HStore(539): loaded hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/72b78dfd0678a920ddc3700612ff3932/cf1/bce8056f97d34999b5a16bbacddae4f5.1c6b8f18205322cbb248ea4fb10155d8->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/cf1/bce8056f97d34999b5a16bbacddae4f5-top 2023-04-12 02:54:27,520 INFO [RS:0;jenkins-hbase4:34579-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 81296e35c0d27ffa1ba7676ff53f87e1/cf1 of 81296e35c0d27ffa1ba7676ff53f87e1 into 072e4ee0f62f45e8ada734db958eabf1(size=6.8 K), total size for store is 6.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 02:54:27,520 DEBUG [RS:0;jenkins-hbase4:34579-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 81296e35c0d27ffa1ba7676ff53f87e1: 2023-04-12 02:54:27,520 INFO [RS:0;jenkins-hbase4:34579-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=testSplitTableRegionDeletedRowsDaughter,,1681268065445.81296e35c0d27ffa1ba7676ff53f87e1., storeName=81296e35c0d27ffa1ba7676ff53f87e1/cf1, priority=9, startTime=1681268067453; duration=0sec 2023-04-12 02:54:27,520 DEBUG [RS:0;jenkins-hbase4:34579-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 02:54:27,523 INFO [StoreFileOpener-cf1-1] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for fdff24c7084643c5a7298e4db0f23442.1c6b8f18205322cbb248ea4fb10155d8 2023-04-12 02:54:27,524 DEBUG [StoreOpener-72b78dfd0678a920ddc3700612ff3932-1] regionserver.HStore(539): loaded hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/72b78dfd0678a920ddc3700612ff3932/cf1/fdff24c7084643c5a7298e4db0f23442.1c6b8f18205322cbb248ea4fb10155d8->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/cf1/fdff24c7084643c5a7298e4db0f23442-top 2023-04-12 02:54:27,525 DEBUG [RS:0;jenkins-hbase4:34579-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/81296e35c0d27ffa1ba7676ff53f87e1/.tmp/cf2/a68a5b61455b4fa0bd7b52420416f066 as hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/81296e35c0d27ffa1ba7676ff53f87e1/cf2/a68a5b61455b4fa0bd7b52420416f066 2023-04-12 02:54:27,542 INFO [StoreFileOpener-cf1-1] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 45831fa527a845c783019394c67c0fa7 2023-04-12 02:54:27,542 DEBUG [StoreOpener-72b78dfd0678a920ddc3700612ff3932-1] regionserver.HStore(539): loaded hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/72b78dfd0678a920ddc3700612ff3932/cf1/testSplitTableRegionDeletedRowsDaughter=1c6b8f18205322cbb248ea4fb10155d8-45831fa527a845c783019394c67c0fa7 2023-04-12 02:54:27,543 INFO [RS:0;jenkins-hbase4:34579-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 81296e35c0d27ffa1ba7676ff53f87e1/cf2 of 81296e35c0d27ffa1ba7676ff53f87e1 into a68a5b61455b4fa0bd7b52420416f066(size=6.8 K), total size for store is 6.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 02:54:27,543 DEBUG [RS:0;jenkins-hbase4:34579-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 81296e35c0d27ffa1ba7676ff53f87e1: 2023-04-12 02:54:27,543 INFO [RS:0;jenkins-hbase4:34579-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=testSplitTableRegionDeletedRowsDaughter,,1681268065445.81296e35c0d27ffa1ba7676ff53f87e1., storeName=81296e35c0d27ffa1ba7676ff53f87e1/cf2, priority=9, startTime=1681268067455; duration=0sec 2023-04-12 02:54:27,543 DEBUG [RS:0;jenkins-hbase4:34579-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 02:54:27,549 INFO [StoreFileOpener-cf1-1] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 4868a2af61a94040a8fcc176f05bb0fc 2023-04-12 02:54:27,549 DEBUG [StoreOpener-72b78dfd0678a920ddc3700612ff3932-1] regionserver.HStore(539): loaded hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/72b78dfd0678a920ddc3700612ff3932/cf1/testSplitTableRegionDeletedRowsDaughter=1c6b8f18205322cbb248ea4fb10155d8-4868a2af61a94040a8fcc176f05bb0fc 2023-04-12 02:54:27,553 INFO [StoreFileOpener-cf1-1] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 85930b8d05664956906c5fa532cccb71 2023-04-12 02:54:27,553 DEBUG [StoreOpener-72b78dfd0678a920ddc3700612ff3932-1] regionserver.HStore(539): loaded hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/72b78dfd0678a920ddc3700612ff3932/cf1/testSplitTableRegionDeletedRowsDaughter=1c6b8f18205322cbb248ea4fb10155d8-85930b8d05664956906c5fa532cccb71 2023-04-12 02:54:27,553 INFO [StoreOpener-72b78dfd0678a920ddc3700612ff3932-1] regionserver.HStore(310): Store=72b78dfd0678a920ddc3700612ff3932/cf1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-12 02:54:27,554 INFO [StoreOpener-72b78dfd0678a920ddc3700612ff3932-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf2 of region 72b78dfd0678a920ddc3700612ff3932 2023-04-12 02:54:27,554 DEBUG [StoreOpener-72b78dfd0678a920ddc3700612ff3932-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/72b78dfd0678a920ddc3700612ff3932/cf2 2023-04-12 02:54:27,554 DEBUG [StoreOpener-72b78dfd0678a920ddc3700612ff3932-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/72b78dfd0678a920ddc3700612ff3932/cf2 2023-04-12 02:54:27,555 INFO [StoreOpener-72b78dfd0678a920ddc3700612ff3932-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 72b78dfd0678a920ddc3700612ff3932 columnFamilyName cf2 2023-04-12 02:54:27,568 DEBUG [StoreOpener-72b78dfd0678a920ddc3700612ff3932-1] regionserver.HStore(539): loaded hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/72b78dfd0678a920ddc3700612ff3932/cf2/381b6116e3b64f31a2b235e9f7067d52.1c6b8f18205322cbb248ea4fb10155d8->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/cf2/381b6116e3b64f31a2b235e9f7067d52-top 2023-04-12 02:54:27,574 DEBUG [StoreOpener-72b78dfd0678a920ddc3700612ff3932-1] regionserver.HStore(539): loaded hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/72b78dfd0678a920ddc3700612ff3932/cf2/77b0532593204cbd90904f2bbec317e7.1c6b8f18205322cbb248ea4fb10155d8->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/cf2/77b0532593204cbd90904f2bbec317e7-top 2023-04-12 02:54:27,579 DEBUG [StoreOpener-72b78dfd0678a920ddc3700612ff3932-1] regionserver.HStore(539): loaded hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/72b78dfd0678a920ddc3700612ff3932/cf2/926ca93402364365a7ac53b2e60b9bf5.1c6b8f18205322cbb248ea4fb10155d8->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/cf2/926ca93402364365a7ac53b2e60b9bf5-top 2023-04-12 02:54:27,583 INFO [StoreFileOpener-cf2-1] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for a01c4e9f61f6405380a578694839ec53.1c6b8f18205322cbb248ea4fb10155d8 2023-04-12 02:54:27,584 DEBUG [StoreOpener-72b78dfd0678a920ddc3700612ff3932-1] regionserver.HStore(539): loaded hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/72b78dfd0678a920ddc3700612ff3932/cf2/a01c4e9f61f6405380a578694839ec53.1c6b8f18205322cbb248ea4fb10155d8->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/cf2/a01c4e9f61f6405380a578694839ec53-top 2023-04-12 02:54:27,588 INFO [StoreFileOpener-cf2-1] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 0c6c01c95a4f4bc3a4312a2eee9e8eec 2023-04-12 02:54:27,588 DEBUG [StoreOpener-72b78dfd0678a920ddc3700612ff3932-1] regionserver.HStore(539): loaded hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/72b78dfd0678a920ddc3700612ff3932/cf2/testSplitTableRegionDeletedRowsDaughter=1c6b8f18205322cbb248ea4fb10155d8-0c6c01c95a4f4bc3a4312a2eee9e8eec 2023-04-12 02:54:27,596 INFO [StoreFileOpener-cf2-1] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 37083e44dea546c8a8710fe5148a6dcd 2023-04-12 02:54:27,596 DEBUG [StoreOpener-72b78dfd0678a920ddc3700612ff3932-1] regionserver.HStore(539): loaded hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/72b78dfd0678a920ddc3700612ff3932/cf2/testSplitTableRegionDeletedRowsDaughter=1c6b8f18205322cbb248ea4fb10155d8-37083e44dea546c8a8710fe5148a6dcd 2023-04-12 02:54:27,600 INFO [StoreFileOpener-cf2-1] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 8ac99bc8ac8f4c3b873212471489db78 2023-04-12 02:54:27,600 DEBUG [StoreOpener-72b78dfd0678a920ddc3700612ff3932-1] regionserver.HStore(539): loaded hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/72b78dfd0678a920ddc3700612ff3932/cf2/testSplitTableRegionDeletedRowsDaughter=1c6b8f18205322cbb248ea4fb10155d8-8ac99bc8ac8f4c3b873212471489db78 2023-04-12 02:54:27,600 INFO [StoreOpener-72b78dfd0678a920ddc3700612ff3932-1] regionserver.HStore(310): Store=72b78dfd0678a920ddc3700612ff3932/cf2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-12 02:54:27,601 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/72b78dfd0678a920ddc3700612ff3932 2023-04-12 02:54:27,602 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/72b78dfd0678a920ddc3700612ff3932 2023-04-12 02:54:27,603 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testSplitTableRegionDeletedRowsDaughter descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-04-12 02:54:27,605 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 72b78dfd0678a920ddc3700612ff3932 2023-04-12 02:54:27,605 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 72b78dfd0678a920ddc3700612ff3932; next sequenceid=107; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=62885824, jitterRate=-0.0629281997680664}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-04-12 02:54:27,606 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 72b78dfd0678a920ddc3700612ff3932: 2023-04-12 02:54:27,606 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testSplitTableRegionDeletedRowsDaughter,60,1681268065445.72b78dfd0678a920ddc3700612ff3932., pid=78, masterSystemTime=1681268067355 2023-04-12 02:54:27,607 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: Opening Region; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 02:54:27,608 DEBUG [RS:0;jenkins-hbase4:34579-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-12 02:54:27,608 DEBUG [RS:0;jenkins-hbase4:34579-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-12 02:54:27,608 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: Opening Region; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 02:54:27,608 INFO [RS:0;jenkins-hbase4:34579-longCompactions-0] regionserver.HStore(1898): Keeping/Overriding Compaction request priority to -2147482648 for CF cf2 since it belongs to recently split daughter region testSplitTableRegionDeletedRowsDaughter,60,1681268065445.72b78dfd0678a920ddc3700612ff3932. 2023-04-12 02:54:27,608 DEBUG [RS:0;jenkins-hbase4:34579-longCompactions-0] regionserver.HStore(1912): 72b78dfd0678a920ddc3700612ff3932/cf2 is initiating minor compaction (all files) 2023-04-12 02:54:27,608 INFO [RS:0;jenkins-hbase4:34579-shortCompactions-0] regionserver.HStore(1898): Keeping/Overriding Compaction request priority to -2147482648 for CF cf1 since it belongs to recently split daughter region testSplitTableRegionDeletedRowsDaughter,60,1681268065445.72b78dfd0678a920ddc3700612ff3932. 2023-04-12 02:54:27,608 INFO [RS:0;jenkins-hbase4:34579-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 72b78dfd0678a920ddc3700612ff3932/cf2 in testSplitTableRegionDeletedRowsDaughter,60,1681268065445.72b78dfd0678a920ddc3700612ff3932. 2023-04-12 02:54:27,608 DEBUG [RS:0;jenkins-hbase4:34579-shortCompactions-0] regionserver.HStore(1912): 72b78dfd0678a920ddc3700612ff3932/cf1 is initiating minor compaction (all files) 2023-04-12 02:54:27,608 INFO [RS:0;jenkins-hbase4:34579-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 72b78dfd0678a920ddc3700612ff3932/cf1 in testSplitTableRegionDeletedRowsDaughter,60,1681268065445.72b78dfd0678a920ddc3700612ff3932. 2023-04-12 02:54:27,608 INFO [RS:0;jenkins-hbase4:34579-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/72b78dfd0678a920ddc3700612ff3932/cf2/926ca93402364365a7ac53b2e60b9bf5.1c6b8f18205322cbb248ea4fb10155d8->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/cf2/926ca93402364365a7ac53b2e60b9bf5-top, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/72b78dfd0678a920ddc3700612ff3932/cf2/77b0532593204cbd90904f2bbec317e7.1c6b8f18205322cbb248ea4fb10155d8->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/cf2/77b0532593204cbd90904f2bbec317e7-top, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/72b78dfd0678a920ddc3700612ff3932/cf2/381b6116e3b64f31a2b235e9f7067d52.1c6b8f18205322cbb248ea4fb10155d8->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/cf2/381b6116e3b64f31a2b235e9f7067d52-top, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/72b78dfd0678a920ddc3700612ff3932/cf2/a01c4e9f61f6405380a578694839ec53.1c6b8f18205322cbb248ea4fb10155d8->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/cf2/a01c4e9f61f6405380a578694839ec53-top, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/72b78dfd0678a920ddc3700612ff3932/cf2/testSplitTableRegionDeletedRowsDaughter=1c6b8f18205322cbb248ea4fb10155d8-0c6c01c95a4f4bc3a4312a2eee9e8eec, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/72b78dfd0678a920ddc3700612ff3932/cf2/testSplitTableRegionDeletedRowsDaughter=1c6b8f18205322cbb248ea4fb10155d8-37083e44dea546c8a8710fe5148a6dcd, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/72b78dfd0678a920ddc3700612ff3932/cf2/testSplitTableRegionDeletedRowsDaughter=1c6b8f18205322cbb248ea4fb10155d8-8ac99bc8ac8f4c3b873212471489db78] into tmpdir=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/72b78dfd0678a920ddc3700612ff3932/.tmp, totalSize=34.8 K 2023-04-12 02:54:27,609 INFO [RS:0;jenkins-hbase4:34579-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/72b78dfd0678a920ddc3700612ff3932/cf1/789e1a943f304f83be48e491529ca4ad.1c6b8f18205322cbb248ea4fb10155d8->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/cf1/789e1a943f304f83be48e491529ca4ad-top, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/72b78dfd0678a920ddc3700612ff3932/cf1/0fd593a3724149b8ad466c2cef9ba846.1c6b8f18205322cbb248ea4fb10155d8->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/cf1/0fd593a3724149b8ad466c2cef9ba846-top, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/72b78dfd0678a920ddc3700612ff3932/cf1/bce8056f97d34999b5a16bbacddae4f5.1c6b8f18205322cbb248ea4fb10155d8->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/cf1/bce8056f97d34999b5a16bbacddae4f5-top, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/72b78dfd0678a920ddc3700612ff3932/cf1/fdff24c7084643c5a7298e4db0f23442.1c6b8f18205322cbb248ea4fb10155d8->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/cf1/fdff24c7084643c5a7298e4db0f23442-top, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/72b78dfd0678a920ddc3700612ff3932/cf1/testSplitTableRegionDeletedRowsDaughter=1c6b8f18205322cbb248ea4fb10155d8-4868a2af61a94040a8fcc176f05bb0fc, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/72b78dfd0678a920ddc3700612ff3932/cf1/testSplitTableRegionDeletedRowsDaughter=1c6b8f18205322cbb248ea4fb10155d8-85930b8d05664956906c5fa532cccb71, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/72b78dfd0678a920ddc3700612ff3932/cf1/testSplitTableRegionDeletedRowsDaughter=1c6b8f18205322cbb248ea4fb10155d8-45831fa527a845c783019394c67c0fa7] into tmpdir=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/72b78dfd0678a920ddc3700612ff3932/.tmp, totalSize=34.8 K 2023-04-12 02:54:27,610 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=76 updating hbase:meta row=72b78dfd0678a920ddc3700612ff3932, regionState=OPEN, openSeqNum=107, regionLocation=jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:27,610 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testSplitTableRegionDeletedRowsDaughter,60,1681268065445.72b78dfd0678a920ddc3700612ff3932.","families":{"info":[{"qualifier":"regioninfo","vlen":75,"tag":[],"timestamp":"1681268067610"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1681268067610"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681268067610"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681268067610"}]},"ts":"1681268067610"} 2023-04-12 02:54:27,610 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testSplitTableRegionDeletedRowsDaughter,60,1681268065445.72b78dfd0678a920ddc3700612ff3932. 2023-04-12 02:54:27,610 DEBUG [RS:0;jenkins-hbase4:34579-shortCompactions-0] compactions.Compactor(207): Compacting 789e1a943f304f83be48e491529ca4ad.1c6b8f18205322cbb248ea4fb10155d8, keycount=1, bloomtype=ROW, size=4.8 K, encoding=NONE, compression=NONE, seqNum=7, earliestPutTs=1681268063181 2023-04-12 02:54:27,611 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testSplitTableRegionDeletedRowsDaughter,60,1681268065445.72b78dfd0678a920ddc3700612ff3932. 2023-04-12 02:54:27,611 DEBUG [RS:0;jenkins-hbase4:34579-longCompactions-0] compactions.Compactor(207): Compacting 926ca93402364365a7ac53b2e60b9bf5.1c6b8f18205322cbb248ea4fb10155d8, keycount=1, bloomtype=ROW, size=4.8 K, encoding=NONE, compression=NONE, seqNum=7, earliestPutTs=1681268063181 2023-04-12 02:54:27,611 DEBUG [RS:0;jenkins-hbase4:34579-shortCompactions-0] compactions.Compactor(207): Compacting 0fd593a3724149b8ad466c2cef9ba846.1c6b8f18205322cbb248ea4fb10155d8, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=20, earliestPutTs=1681268063370 2023-04-12 02:54:27,612 DEBUG [RS:0;jenkins-hbase4:34579-longCompactions-0] compactions.Compactor(207): Compacting 77b0532593204cbd90904f2bbec317e7.1c6b8f18205322cbb248ea4fb10155d8, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=20, earliestPutTs=1681268063370 2023-04-12 02:54:27,612 DEBUG [RS:0;jenkins-hbase4:34579-shortCompactions-0] compactions.Compactor(207): Compacting bce8056f97d34999b5a16bbacddae4f5.1c6b8f18205322cbb248ea4fb10155d8, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=33, earliestPutTs=1681268063570 2023-04-12 02:54:27,612 DEBUG [RS:0;jenkins-hbase4:34579-longCompactions-0] compactions.Compactor(207): Compacting 381b6116e3b64f31a2b235e9f7067d52.1c6b8f18205322cbb248ea4fb10155d8, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=33, earliestPutTs=1681268063570 2023-04-12 02:54:27,613 DEBUG [RS:0;jenkins-hbase4:34579-shortCompactions-0] compactions.Compactor(207): Compacting fdff24c7084643c5a7298e4db0f23442.1c6b8f18205322cbb248ea4fb10155d8, keycount=4, bloomtype=ROW, size=5.1 K, encoding=NONE, compression=NONE, seqNum=84, earliestPutTs=1681268064634 2023-04-12 02:54:27,613 DEBUG [RS:0;jenkins-hbase4:34579-longCompactions-0] compactions.Compactor(207): Compacting a01c4e9f61f6405380a578694839ec53.1c6b8f18205322cbb248ea4fb10155d8, keycount=4, bloomtype=ROW, size=5.1 K, encoding=NONE, compression=NONE, seqNum=84, earliestPutTs=1681268064634 2023-04-12 02:54:27,614 DEBUG [RS:0;jenkins-hbase4:34579-shortCompactions-0] compactions.Compactor(207): Compacting testSplitTableRegionDeletedRowsDaughter=1c6b8f18205322cbb248ea4fb10155d8-4868a2af61a94040a8fcc176f05bb0fc, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=91, earliestPutTs=9223372036854775807 2023-04-12 02:54:27,614 DEBUG [RS:0;jenkins-hbase4:34579-longCompactions-0] compactions.Compactor(207): Compacting testSplitTableRegionDeletedRowsDaughter=1c6b8f18205322cbb248ea4fb10155d8-0c6c01c95a4f4bc3a4312a2eee9e8eec, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=91, earliestPutTs=9223372036854775807 2023-04-12 02:54:27,614 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=78, resume processing ppid=76 2023-04-12 02:54:27,614 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=78, ppid=76, state=SUCCESS; OpenRegionProcedure 72b78dfd0678a920ddc3700612ff3932, server=jenkins-hbase4.apache.org,34579,1681268031719 in 403 msec 2023-04-12 02:54:27,614 DEBUG [RS:0;jenkins-hbase4:34579-shortCompactions-0] compactions.Compactor(207): Compacting testSplitTableRegionDeletedRowsDaughter=1c6b8f18205322cbb248ea4fb10155d8-85930b8d05664956906c5fa532cccb71, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=99, earliestPutTs=9223372036854775807 2023-04-12 02:54:27,614 DEBUG [RS:0;jenkins-hbase4:34579-longCompactions-0] compactions.Compactor(207): Compacting testSplitTableRegionDeletedRowsDaughter=1c6b8f18205322cbb248ea4fb10155d8-37083e44dea546c8a8710fe5148a6dcd, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=99, earliestPutTs=9223372036854775807 2023-04-12 02:54:27,614 DEBUG [RS:0;jenkins-hbase4:34579-shortCompactions-0] compactions.Compactor(207): Compacting testSplitTableRegionDeletedRowsDaughter=1c6b8f18205322cbb248ea4fb10155d8-45831fa527a845c783019394c67c0fa7, keycount=1, bloomtype=ROW, size=4.9 K, encoding=NONE, compression=NONE, seqNum=103, earliestPutTs=9223372036854775807 2023-04-12 02:54:27,615 DEBUG [RS:0;jenkins-hbase4:34579-longCompactions-0] compactions.Compactor(207): Compacting testSplitTableRegionDeletedRowsDaughter=1c6b8f18205322cbb248ea4fb10155d8-8ac99bc8ac8f4c3b873212471489db78, keycount=1, bloomtype=ROW, size=4.9 K, encoding=NONE, compression=NONE, seqNum=103, earliestPutTs=9223372036854775807 2023-04-12 02:54:27,616 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=76, resume processing ppid=72 2023-04-12 02:54:27,617 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=76, ppid=72, state=SUCCESS; TransitRegionStateProcedure table=testSplitTableRegionDeletedRowsDaughter, region=72b78dfd0678a920ddc3700612ff3932, ASSIGN in 567 msec 2023-04-12 02:54:27,619 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=72, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionDeletedRowsDaughter, parent=1c6b8f18205322cbb248ea4fb10155d8, daughterA=81296e35c0d27ffa1ba7676ff53f87e1, daughterB=72b78dfd0678a920ddc3700612ff3932 in 2.1720 sec 2023-04-12 02:54:27,656 INFO [RS:0;jenkins-hbase4:34579-longCompactions-0] throttle.PressureAwareThroughputController(145): 72b78dfd0678a920ddc3700612ff3932#cf2#compaction#74 average throughput is 0.00 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 02:54:27,677 INFO [RS:0;jenkins-hbase4:34579-shortCompactions-0] throttle.PressureAwareThroughputController(145): 72b78dfd0678a920ddc3700612ff3932#cf1#compaction#75 average throughput is 0.00 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 02:54:27,698 DEBUG [RS:0;jenkins-hbase4:34579-longCompactions-0] regionserver.HStoreFile(437): HFile Bloom filter type for 96fe5b19e4694f46927e13df8f2c6a66: NONE, but ROW specified in column family configuration 2023-04-12 02:54:27,699 DEBUG [RS:0;jenkins-hbase4:34579-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/72b78dfd0678a920ddc3700612ff3932/.tmp/cf2/96fe5b19e4694f46927e13df8f2c6a66 as hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/72b78dfd0678a920ddc3700612ff3932/cf2/96fe5b19e4694f46927e13df8f2c6a66 2023-04-12 02:54:27,704 DEBUG [RS:0;jenkins-hbase4:34579-longCompactions-0] regionserver.HStoreFile(437): HFile Bloom filter type for 96fe5b19e4694f46927e13df8f2c6a66: NONE, but ROW specified in column family configuration 2023-04-12 02:54:27,705 INFO [RS:0;jenkins-hbase4:34579-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 72b78dfd0678a920ddc3700612ff3932/cf2 of 72b78dfd0678a920ddc3700612ff3932 into 96fe5b19e4694f46927e13df8f2c6a66(size=5.0 K), total size for store is 5.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 02:54:27,705 DEBUG [RS:0;jenkins-hbase4:34579-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 72b78dfd0678a920ddc3700612ff3932: 2023-04-12 02:54:27,705 INFO [RS:0;jenkins-hbase4:34579-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=testSplitTableRegionDeletedRowsDaughter,60,1681268065445.72b78dfd0678a920ddc3700612ff3932., storeName=72b78dfd0678a920ddc3700612ff3932/cf2, priority=9, startTime=1681268067608; duration=0sec 2023-04-12 02:54:27,705 DEBUG [RS:0;jenkins-hbase4:34579-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 02:54:27,715 DEBUG [RS:0;jenkins-hbase4:34579-shortCompactions-0] regionserver.HStoreFile(437): HFile Bloom filter type for f5d7697e71da4fea98fd0205d2d3a52b: NONE, but ROW specified in column family configuration 2023-04-12 02:54:27,716 DEBUG [RS:0;jenkins-hbase4:34579-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/72b78dfd0678a920ddc3700612ff3932/.tmp/cf1/f5d7697e71da4fea98fd0205d2d3a52b as hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/72b78dfd0678a920ddc3700612ff3932/cf1/f5d7697e71da4fea98fd0205d2d3a52b 2023-04-12 02:54:27,720 DEBUG [RS:0;jenkins-hbase4:34579-shortCompactions-0] regionserver.HStoreFile(437): HFile Bloom filter type for f5d7697e71da4fea98fd0205d2d3a52b: NONE, but ROW specified in column family configuration 2023-04-12 02:54:27,721 INFO [RS:0;jenkins-hbase4:34579-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 72b78dfd0678a920ddc3700612ff3932/cf1 of 72b78dfd0678a920ddc3700612ff3932 into f5d7697e71da4fea98fd0205d2d3a52b(size=5.0 K), total size for store is 5.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 02:54:27,721 DEBUG [RS:0;jenkins-hbase4:34579-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 72b78dfd0678a920ddc3700612ff3932: 2023-04-12 02:54:27,721 INFO [RS:0;jenkins-hbase4:34579-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=testSplitTableRegionDeletedRowsDaughter,60,1681268065445.72b78dfd0678a920ddc3700612ff3932., storeName=72b78dfd0678a920ddc3700612ff3932/cf1, priority=9, startTime=1681268067607; duration=0sec 2023-04-12 02:54:27,722 DEBUG [RS:0;jenkins-hbase4:34579-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 02:54:27,724 DEBUG [Listener at localhost/46667] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-04-12 02:54:27,727 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:44214, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-04-12 02:54:27,727 INFO [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=34579] regionserver.RSRpcServices(1662): Compacting testSplitTableRegionDeletedRowsDaughter,,1681268065445.81296e35c0d27ffa1ba7676ff53f87e1. 2023-04-12 02:54:27,729 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=34579] compactions.SortedCompactionPolicy(75): Selecting compaction from 1 store files, 0 compacting, 1 eligible, 16 blocking 2023-04-12 02:54:27,729 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=34579] regionserver.HStore(1912): 81296e35c0d27ffa1ba7676ff53f87e1/cf1 is initiating major compaction (all files) 2023-04-12 02:54:27,729 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=34579] regionserver.CompactSplit(385): Small Compaction requested: org.apache.hadoop.hbase.regionserver.DefaultStoreEngine$DefaultCompactionContext@66b7f39; Because: User-triggered major compaction for region testSplitTableRegionDeletedRowsDaughter,,1681268065445.81296e35c0d27ffa1ba7676ff53f87e1.; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 02:54:27,729 INFO [RS:0;jenkins-hbase4:34579-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 81296e35c0d27ffa1ba7676ff53f87e1/cf1 in testSplitTableRegionDeletedRowsDaughter,,1681268065445.81296e35c0d27ffa1ba7676ff53f87e1. 2023-04-12 02:54:27,729 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=34579] compactions.SortedCompactionPolicy(75): Selecting compaction from 1 store files, 0 compacting, 1 eligible, 16 blocking 2023-04-12 02:54:27,729 INFO [RS:0;jenkins-hbase4:34579-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/81296e35c0d27ffa1ba7676ff53f87e1/cf1/072e4ee0f62f45e8ada734db958eabf1] into tmpdir=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/81296e35c0d27ffa1ba7676ff53f87e1/.tmp, totalSize=6.8 K 2023-04-12 02:54:27,729 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=34579] regionserver.HStore(1912): 81296e35c0d27ffa1ba7676ff53f87e1/cf2 is initiating major compaction (all files) 2023-04-12 02:54:27,729 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=34579] regionserver.CompactSplit(385): Small Compaction requested: org.apache.hadoop.hbase.regionserver.DefaultStoreEngine$DefaultCompactionContext@43391663; Because: User-triggered major compaction for region testSplitTableRegionDeletedRowsDaughter,,1681268065445.81296e35c0d27ffa1ba7676ff53f87e1.; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 02:54:27,730 INFO [RS:0;jenkins-hbase4:34579-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 81296e35c0d27ffa1ba7676ff53f87e1/cf2 in testSplitTableRegionDeletedRowsDaughter,,1681268065445.81296e35c0d27ffa1ba7676ff53f87e1. 2023-04-12 02:54:27,730 INFO [RS:0;jenkins-hbase4:34579-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/81296e35c0d27ffa1ba7676ff53f87e1/cf2/a68a5b61455b4fa0bd7b52420416f066] into tmpdir=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/81296e35c0d27ffa1ba7676ff53f87e1/.tmp, totalSize=6.8 K 2023-04-12 02:54:27,731 DEBUG [RS:0;jenkins-hbase4:34579-longCompactions-0] compactions.Compactor(207): Compacting 072e4ee0f62f45e8ada734db958eabf1, keycount=49, bloomtype=ROW, size=6.8 K, encoding=NONE, compression=NONE, seqNum=83, earliestPutTs=1681268063181 2023-04-12 02:54:27,732 DEBUG [RS:0;jenkins-hbase4:34579-shortCompactions-0] compactions.Compactor(207): Compacting a68a5b61455b4fa0bd7b52420416f066, keycount=49, bloomtype=ROW, size=6.8 K, encoding=NONE, compression=NONE, seqNum=83, earliestPutTs=1681268063181 2023-04-12 02:54:27,732 INFO [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=34579] regionserver.RSRpcServices(1662): Compacting testSplitTableRegionDeletedRowsDaughter,60,1681268065445.72b78dfd0678a920ddc3700612ff3932. 2023-04-12 02:54:27,732 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=34579] compactions.SortedCompactionPolicy(75): Selecting compaction from 1 store files, 0 compacting, 1 eligible, 16 blocking 2023-04-12 02:54:27,732 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=34579] regionserver.HStore(1912): 72b78dfd0678a920ddc3700612ff3932/cf1 is initiating major compaction (all files) 2023-04-12 02:54:27,732 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=34579] regionserver.CompactSplit(385): Small Compaction requested: org.apache.hadoop.hbase.regionserver.DefaultStoreEngine$DefaultCompactionContext@7b02a365; Because: User-triggered major compaction for region testSplitTableRegionDeletedRowsDaughter,60,1681268065445.72b78dfd0678a920ddc3700612ff3932.; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 02:54:27,732 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=34579] compactions.SortedCompactionPolicy(75): Selecting compaction from 1 store files, 0 compacting, 1 eligible, 16 blocking 2023-04-12 02:54:27,733 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=34579] regionserver.HStore(1912): 72b78dfd0678a920ddc3700612ff3932/cf2 is initiating major compaction (all files) 2023-04-12 02:54:27,733 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=34579] regionserver.CompactSplit(385): Small Compaction requested: org.apache.hadoop.hbase.regionserver.DefaultStoreEngine$DefaultCompactionContext@2af9935e; Because: User-triggered major compaction for region testSplitTableRegionDeletedRowsDaughter,60,1681268065445.72b78dfd0678a920ddc3700612ff3932.; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-12 02:54:27,734 INFO [Listener at localhost/46667] hbase.Waiter(180): Waiting up to [6,000] milli-secs(wait.for.ratio=[1]) 2023-04-12 02:54:27,756 INFO [RS:0;jenkins-hbase4:34579-longCompactions-0] throttle.PressureAwareThroughputController(145): 81296e35c0d27ffa1ba7676ff53f87e1#cf1#compaction#76 average throughput is 1.40 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 02:54:27,762 INFO [RS:0;jenkins-hbase4:34579-shortCompactions-0] throttle.PressureAwareThroughputController(145): 81296e35c0d27ffa1ba7676ff53f87e1#cf2#compaction#77 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 02:54:27,790 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=34579] regionserver.HRegion(1451): Region testSplitTableRegionDeletedRowsDaughter,,1681268065445.81296e35c0d27ffa1ba7676ff53f87e1. is not mergeable because it has references 2023-04-12 02:54:27,791 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=34579] regionserver.HRegion(1451): Region testSplitTableRegionDeletedRowsDaughter,60,1681268065445.72b78dfd0678a920ddc3700612ff3932. is not mergeable because it has references 2023-04-12 02:54:27,861 DEBUG [RS:0;jenkins-hbase4:34579-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/81296e35c0d27ffa1ba7676ff53f87e1/.tmp/cf1/9739f85c05884e7bbee8b2cf14b73c87 as hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/81296e35c0d27ffa1ba7676ff53f87e1/cf1/9739f85c05884e7bbee8b2cf14b73c87 2023-04-12 02:54:27,867 INFO [RS:0;jenkins-hbase4:34579-longCompactions-0] regionserver.HStore(1652): Completed major compaction of 1 (all) file(s) in 81296e35c0d27ffa1ba7676ff53f87e1/cf1 of 81296e35c0d27ffa1ba7676ff53f87e1 into 9739f85c05884e7bbee8b2cf14b73c87(size=6.9 K), total size for store is 6.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 02:54:27,867 DEBUG [RS:0;jenkins-hbase4:34579-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 81296e35c0d27ffa1ba7676ff53f87e1: 2023-04-12 02:54:27,869 INFO [RS:0;jenkins-hbase4:34579-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction Request=regionName=testSplitTableRegionDeletedRowsDaughter,,1681268065445.81296e35c0d27ffa1ba7676ff53f87e1., storeName=cf1, fileCount=1, fileSize=6.8 K (6.8 K), priority=1, time=1681268067729; duration=0sec 2023-04-12 02:54:27,869 DEBUG [RS:0;jenkins-hbase4:34579-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-12 02:54:27,869 INFO [RS:0;jenkins-hbase4:34579-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 72b78dfd0678a920ddc3700612ff3932/cf1 in testSplitTableRegionDeletedRowsDaughter,60,1681268065445.72b78dfd0678a920ddc3700612ff3932. 2023-04-12 02:54:27,869 INFO [RS:0;jenkins-hbase4:34579-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/72b78dfd0678a920ddc3700612ff3932/cf1/f5d7697e71da4fea98fd0205d2d3a52b] into tmpdir=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/72b78dfd0678a920ddc3700612ff3932/.tmp, totalSize=5.0 K 2023-04-12 02:54:27,870 DEBUG [RS:0;jenkins-hbase4:34579-longCompactions-0] compactions.Compactor(207): Compacting f5d7697e71da4fea98fd0205d2d3a52b, keycount=0, bloomtype=NONE, size=5.0 K, encoding=NONE, compression=NONE, seqNum=103, earliestPutTs=9223372036854775807 2023-04-12 02:54:27,894 INFO [RS:0;jenkins-hbase4:34579-longCompactions-0] throttle.PressureAwareThroughputController(145): 72b78dfd0678a920ddc3700612ff3932#cf1#compaction#78 average throughput is NaN MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 02:54:27,949 DEBUG [RS:0;jenkins-hbase4:34579-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/81296e35c0d27ffa1ba7676ff53f87e1/.tmp/cf2/4a5dbb079f0845959e0cd2c5ba7122bf as hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/81296e35c0d27ffa1ba7676ff53f87e1/cf2/4a5dbb079f0845959e0cd2c5ba7122bf 2023-04-12 02:54:27,956 INFO [RS:0;jenkins-hbase4:34579-shortCompactions-0] regionserver.HStore(1652): Completed major compaction of 1 (all) file(s) in 81296e35c0d27ffa1ba7676ff53f87e1/cf2 of 81296e35c0d27ffa1ba7676ff53f87e1 into 4a5dbb079f0845959e0cd2c5ba7122bf(size=6.9 K), total size for store is 6.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 02:54:27,956 DEBUG [RS:0;jenkins-hbase4:34579-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 81296e35c0d27ffa1ba7676ff53f87e1: 2023-04-12 02:54:27,956 INFO [RS:0;jenkins-hbase4:34579-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction Request=regionName=testSplitTableRegionDeletedRowsDaughter,,1681268065445.81296e35c0d27ffa1ba7676ff53f87e1., storeName=cf2, fileCount=1, fileSize=6.8 K (6.8 K), priority=1, time=1681268067729; duration=0sec 2023-04-12 02:54:27,956 DEBUG [RS:0;jenkins-hbase4:34579-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 02:54:27,956 INFO [RS:0;jenkins-hbase4:34579-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 72b78dfd0678a920ddc3700612ff3932/cf2 in testSplitTableRegionDeletedRowsDaughter,60,1681268065445.72b78dfd0678a920ddc3700612ff3932. 2023-04-12 02:54:27,956 INFO [RS:0;jenkins-hbase4:34579-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/72b78dfd0678a920ddc3700612ff3932/cf2/96fe5b19e4694f46927e13df8f2c6a66] into tmpdir=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/72b78dfd0678a920ddc3700612ff3932/.tmp, totalSize=5.0 K 2023-04-12 02:54:27,959 DEBUG [RS:0;jenkins-hbase4:34579-shortCompactions-0] compactions.Compactor(207): Compacting 96fe5b19e4694f46927e13df8f2c6a66, keycount=0, bloomtype=NONE, size=5.0 K, encoding=NONE, compression=NONE, seqNum=103, earliestPutTs=9223372036854775807 2023-04-12 02:54:27,966 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=34579] regionserver.HRegion(1451): Region testSplitTableRegionDeletedRowsDaughter,,1681268065445.81296e35c0d27ffa1ba7676ff53f87e1. is not mergeable because it has references 2023-04-12 02:54:27,967 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=34579] regionserver.HRegion(1451): Region testSplitTableRegionDeletedRowsDaughter,60,1681268065445.72b78dfd0678a920ddc3700612ff3932. is not mergeable because it has references 2023-04-12 02:54:27,972 INFO [RS:0;jenkins-hbase4:34579-shortCompactions-0] throttle.PressureAwareThroughputController(145): 72b78dfd0678a920ddc3700612ff3932#cf2#compaction#79 average throughput is NaN MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 02:54:27,982 DEBUG [RS:0;jenkins-hbase4:34579-longCompactions-0] regionserver.HStoreFile(437): HFile Bloom filter type for 95b6df5c670c47fdbb882ac67b11ad32: NONE, but ROW specified in column family configuration 2023-04-12 02:54:27,988 DEBUG [RS:0;jenkins-hbase4:34579-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/72b78dfd0678a920ddc3700612ff3932/.tmp/cf1/95b6df5c670c47fdbb882ac67b11ad32 as hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/72b78dfd0678a920ddc3700612ff3932/cf1/95b6df5c670c47fdbb882ac67b11ad32 2023-04-12 02:54:27,992 DEBUG [RS:0;jenkins-hbase4:34579-longCompactions-0] regionserver.HStoreFile(437): HFile Bloom filter type for 95b6df5c670c47fdbb882ac67b11ad32: NONE, but ROW specified in column family configuration 2023-04-12 02:54:27,994 INFO [RS:0;jenkins-hbase4:34579-longCompactions-0] regionserver.HStore(1652): Completed major compaction of 1 (all) file(s) in 72b78dfd0678a920ddc3700612ff3932/cf1 of 72b78dfd0678a920ddc3700612ff3932 into 95b6df5c670c47fdbb882ac67b11ad32(size=5.0 K), total size for store is 5.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 02:54:27,994 DEBUG [RS:0;jenkins-hbase4:34579-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 72b78dfd0678a920ddc3700612ff3932: 2023-04-12 02:54:27,994 INFO [RS:0;jenkins-hbase4:34579-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction Request=regionName=testSplitTableRegionDeletedRowsDaughter,60,1681268065445.72b78dfd0678a920ddc3700612ff3932., storeName=cf1, fileCount=1, fileSize=5.0 K (5.0 K), priority=1, time=1681268067732; duration=0sec 2023-04-12 02:54:27,994 DEBUG [RS:0;jenkins-hbase4:34579-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 02:54:28,032 DEBUG [RS:0;jenkins-hbase4:34579-shortCompactions-0] regionserver.HStoreFile(437): HFile Bloom filter type for 069237fd2acc4ec79a042514c86be2bf: NONE, but ROW specified in column family configuration 2023-04-12 02:54:28,032 DEBUG [RS:0;jenkins-hbase4:34579-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/72b78dfd0678a920ddc3700612ff3932/.tmp/cf2/069237fd2acc4ec79a042514c86be2bf as hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/72b78dfd0678a920ddc3700612ff3932/cf2/069237fd2acc4ec79a042514c86be2bf 2023-04-12 02:54:28,037 DEBUG [RS:0;jenkins-hbase4:34579-shortCompactions-0] regionserver.HStoreFile(437): HFile Bloom filter type for 069237fd2acc4ec79a042514c86be2bf: NONE, but ROW specified in column family configuration 2023-04-12 02:54:28,039 INFO [RS:0;jenkins-hbase4:34579-shortCompactions-0] regionserver.HStore(1652): Completed major compaction of 1 (all) file(s) in 72b78dfd0678a920ddc3700612ff3932/cf2 of 72b78dfd0678a920ddc3700612ff3932 into 069237fd2acc4ec79a042514c86be2bf(size=5.0 K), total size for store is 5.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 02:54:28,039 DEBUG [RS:0;jenkins-hbase4:34579-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 72b78dfd0678a920ddc3700612ff3932: 2023-04-12 02:54:28,039 INFO [RS:0;jenkins-hbase4:34579-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction Request=regionName=testSplitTableRegionDeletedRowsDaughter,60,1681268065445.72b78dfd0678a920ddc3700612ff3932., storeName=cf2, fileCount=1, fileSize=5.0 K (5.0 K), priority=1, time=1681268067733; duration=0sec 2023-04-12 02:54:28,039 DEBUG [RS:0;jenkins-hbase4:34579-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 02:54:28,076 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=34579] regionserver.HRegion(1451): Region testSplitTableRegionDeletedRowsDaughter,,1681268065445.81296e35c0d27ffa1ba7676ff53f87e1. is not mergeable because it has references 2023-04-12 02:54:28,077 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=34579] regionserver.HRegion(1451): Region testSplitTableRegionDeletedRowsDaughter,60,1681268065445.72b78dfd0678a920ddc3700612ff3932. is not mergeable because it has references 2023-04-12 02:54:28,092 DEBUG [Listener at localhost/46667] client.ClientScanner(240): Advancing internal scanner to startKey at '60', inclusive 2023-04-12 02:54:28,094 WARN [Listener at localhost/46667] procedure2.ProcedureTestingUtility(193): Set Kill before store update to: false 2023-04-12 02:54:28,096 INFO [Listener at localhost/46667] client.HBaseAdmin$15(890): Started disable of testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:28,096 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:28,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] procedure2.ProcedureExecutor(1029): Stored pid=79, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:28,104 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testSplitTableRegionDeletedRowsDaughter","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681268068104"}]},"ts":"1681268068104"} 2023-04-12 02:54:28,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] master.MasterRpcServices(1227): Checking to see if procedure is done pid=79 2023-04-12 02:54:28,108 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testSplitTableRegionDeletedRowsDaughter, state=DISABLING in hbase:meta 2023-04-12 02:54:28,110 INFO [PEWorker-1] procedure.DisableTableProcedure(293): Set testSplitTableRegionDeletedRowsDaughter to state=DISABLING 2023-04-12 02:54:28,112 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=80, ppid=79, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitTableRegionDeletedRowsDaughter, region=81296e35c0d27ffa1ba7676ff53f87e1, UNASSIGN}, {pid=81, ppid=79, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitTableRegionDeletedRowsDaughter, region=72b78dfd0678a920ddc3700612ff3932, UNASSIGN}] 2023-04-12 02:54:28,113 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=81, ppid=79, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitTableRegionDeletedRowsDaughter, region=72b78dfd0678a920ddc3700612ff3932, UNASSIGN 2023-04-12 02:54:28,115 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=81 updating hbase:meta row=72b78dfd0678a920ddc3700612ff3932, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:28,115 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitTableRegionDeletedRowsDaughter,60,1681268065445.72b78dfd0678a920ddc3700612ff3932.","families":{"info":[{"qualifier":"regioninfo","vlen":75,"tag":[],"timestamp":"1681268068114"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681268068114"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681268068114"}]},"ts":"1681268068114"} 2023-04-12 02:54:28,116 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=82, ppid=81, state=RUNNABLE; CloseRegionProcedure 72b78dfd0678a920ddc3700612ff3932, server=jenkins-hbase4.apache.org,34579,1681268031719}] 2023-04-12 02:54:28,120 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=80, ppid=79, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitTableRegionDeletedRowsDaughter, region=81296e35c0d27ffa1ba7676ff53f87e1, UNASSIGN 2023-04-12 02:54:28,121 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=80 updating hbase:meta row=81296e35c0d27ffa1ba7676ff53f87e1, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:28,121 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitTableRegionDeletedRowsDaughter,,1681268065445.81296e35c0d27ffa1ba7676ff53f87e1.","families":{"info":[{"qualifier":"regioninfo","vlen":75,"tag":[],"timestamp":"1681268068120"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681268068120"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681268068120"}]},"ts":"1681268068120"} 2023-04-12 02:54:28,124 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=83, ppid=80, state=RUNNABLE; CloseRegionProcedure 81296e35c0d27ffa1ba7676ff53f87e1, server=jenkins-hbase4.apache.org,34579,1681268031719}] 2023-04-12 02:54:28,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] master.MasterRpcServices(1227): Checking to see if procedure is done pid=79 2023-04-12 02:54:28,269 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 81296e35c0d27ffa1ba7676ff53f87e1 2023-04-12 02:54:28,269 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 81296e35c0d27ffa1ba7676ff53f87e1, disabling compactions & flushes 2023-04-12 02:54:28,269 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region testSplitTableRegionDeletedRowsDaughter,,1681268065445.81296e35c0d27ffa1ba7676ff53f87e1. 2023-04-12 02:54:28,270 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testSplitTableRegionDeletedRowsDaughter,,1681268065445.81296e35c0d27ffa1ba7676ff53f87e1. 2023-04-12 02:54:28,270 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on testSplitTableRegionDeletedRowsDaughter,,1681268065445.81296e35c0d27ffa1ba7676ff53f87e1. after waiting 0 ms 2023-04-12 02:54:28,270 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region testSplitTableRegionDeletedRowsDaughter,,1681268065445.81296e35c0d27ffa1ba7676ff53f87e1. 2023-04-12 02:54:28,270 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,,1681268065445.81296e35c0d27ffa1ba7676ff53f87e1.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/81296e35c0d27ffa1ba7676ff53f87e1/cf1/789e1a943f304f83be48e491529ca4ad.1c6b8f18205322cbb248ea4fb10155d8->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/cf1/789e1a943f304f83be48e491529ca4ad-bottom, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/81296e35c0d27ffa1ba7676ff53f87e1/cf1/0fd593a3724149b8ad466c2cef9ba846.1c6b8f18205322cbb248ea4fb10155d8->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/cf1/0fd593a3724149b8ad466c2cef9ba846-bottom, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/81296e35c0d27ffa1ba7676ff53f87e1/cf1/bce8056f97d34999b5a16bbacddae4f5.1c6b8f18205322cbb248ea4fb10155d8->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/cf1/bce8056f97d34999b5a16bbacddae4f5-bottom, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/81296e35c0d27ffa1ba7676ff53f87e1/cf1/testSplitTableRegionDeletedRowsDaughter=1c6b8f18205322cbb248ea4fb10155d8-985388b30d784536a1f270feb7f52366, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/81296e35c0d27ffa1ba7676ff53f87e1/cf1/testSplitTableRegionDeletedRowsDaughter=1c6b8f18205322cbb248ea4fb10155d8-710424cef2c64d2ebc740c81294a95cb, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/81296e35c0d27ffa1ba7676ff53f87e1/cf1/testSplitTableRegionDeletedRowsDaughter=1c6b8f18205322cbb248ea4fb10155d8-9c4efc02782d420291d1fe480cade0d3, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/81296e35c0d27ffa1ba7676ff53f87e1/cf1/072e4ee0f62f45e8ada734db958eabf1, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/81296e35c0d27ffa1ba7676ff53f87e1/cf1/fdff24c7084643c5a7298e4db0f23442.1c6b8f18205322cbb248ea4fb10155d8->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/cf1/fdff24c7084643c5a7298e4db0f23442-bottom] to archive 2023-04-12 02:54:28,271 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,,1681268065445.81296e35c0d27ffa1ba7676ff53f87e1.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-12 02:54:28,273 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,,1681268065445.81296e35c0d27ffa1ba7676ff53f87e1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/81296e35c0d27ffa1ba7676ff53f87e1/cf1/789e1a943f304f83be48e491529ca4ad.1c6b8f18205322cbb248ea4fb10155d8 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegionDeletedRowsDaughter/81296e35c0d27ffa1ba7676ff53f87e1/cf1/789e1a943f304f83be48e491529ca4ad.1c6b8f18205322cbb248ea4fb10155d8 2023-04-12 02:54:28,274 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,,1681268065445.81296e35c0d27ffa1ba7676ff53f87e1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/81296e35c0d27ffa1ba7676ff53f87e1/cf1/0fd593a3724149b8ad466c2cef9ba846.1c6b8f18205322cbb248ea4fb10155d8 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegionDeletedRowsDaughter/81296e35c0d27ffa1ba7676ff53f87e1/cf1/0fd593a3724149b8ad466c2cef9ba846.1c6b8f18205322cbb248ea4fb10155d8 2023-04-12 02:54:28,275 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,,1681268065445.81296e35c0d27ffa1ba7676ff53f87e1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/81296e35c0d27ffa1ba7676ff53f87e1/cf1/bce8056f97d34999b5a16bbacddae4f5.1c6b8f18205322cbb248ea4fb10155d8 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegionDeletedRowsDaughter/81296e35c0d27ffa1ba7676ff53f87e1/cf1/bce8056f97d34999b5a16bbacddae4f5.1c6b8f18205322cbb248ea4fb10155d8 2023-04-12 02:54:28,276 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,,1681268065445.81296e35c0d27ffa1ba7676ff53f87e1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/81296e35c0d27ffa1ba7676ff53f87e1/cf1/testSplitTableRegionDeletedRowsDaughter=1c6b8f18205322cbb248ea4fb10155d8-985388b30d784536a1f270feb7f52366 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegionDeletedRowsDaughter/81296e35c0d27ffa1ba7676ff53f87e1/cf1/testSplitTableRegionDeletedRowsDaughter=1c6b8f18205322cbb248ea4fb10155d8-985388b30d784536a1f270feb7f52366 2023-04-12 02:54:28,278 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,,1681268065445.81296e35c0d27ffa1ba7676ff53f87e1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/81296e35c0d27ffa1ba7676ff53f87e1/cf1/testSplitTableRegionDeletedRowsDaughter=1c6b8f18205322cbb248ea4fb10155d8-710424cef2c64d2ebc740c81294a95cb to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegionDeletedRowsDaughter/81296e35c0d27ffa1ba7676ff53f87e1/cf1/testSplitTableRegionDeletedRowsDaughter=1c6b8f18205322cbb248ea4fb10155d8-710424cef2c64d2ebc740c81294a95cb 2023-04-12 02:54:28,279 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,,1681268065445.81296e35c0d27ffa1ba7676ff53f87e1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/81296e35c0d27ffa1ba7676ff53f87e1/cf1/testSplitTableRegionDeletedRowsDaughter=1c6b8f18205322cbb248ea4fb10155d8-9c4efc02782d420291d1fe480cade0d3 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegionDeletedRowsDaughter/81296e35c0d27ffa1ba7676ff53f87e1/cf1/testSplitTableRegionDeletedRowsDaughter=1c6b8f18205322cbb248ea4fb10155d8-9c4efc02782d420291d1fe480cade0d3 2023-04-12 02:54:28,280 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,,1681268065445.81296e35c0d27ffa1ba7676ff53f87e1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/81296e35c0d27ffa1ba7676ff53f87e1/cf1/072e4ee0f62f45e8ada734db958eabf1 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegionDeletedRowsDaughter/81296e35c0d27ffa1ba7676ff53f87e1/cf1/072e4ee0f62f45e8ada734db958eabf1 2023-04-12 02:54:28,281 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,,1681268065445.81296e35c0d27ffa1ba7676ff53f87e1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/81296e35c0d27ffa1ba7676ff53f87e1/cf1/fdff24c7084643c5a7298e4db0f23442.1c6b8f18205322cbb248ea4fb10155d8 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegionDeletedRowsDaughter/81296e35c0d27ffa1ba7676ff53f87e1/cf1/fdff24c7084643c5a7298e4db0f23442.1c6b8f18205322cbb248ea4fb10155d8 2023-04-12 02:54:28,284 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,,1681268065445.81296e35c0d27ffa1ba7676ff53f87e1.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/81296e35c0d27ffa1ba7676ff53f87e1/cf2/926ca93402364365a7ac53b2e60b9bf5.1c6b8f18205322cbb248ea4fb10155d8->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/cf2/926ca93402364365a7ac53b2e60b9bf5-bottom, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/81296e35c0d27ffa1ba7676ff53f87e1/cf2/77b0532593204cbd90904f2bbec317e7.1c6b8f18205322cbb248ea4fb10155d8->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/cf2/77b0532593204cbd90904f2bbec317e7-bottom, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/81296e35c0d27ffa1ba7676ff53f87e1/cf2/381b6116e3b64f31a2b235e9f7067d52.1c6b8f18205322cbb248ea4fb10155d8->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/cf2/381b6116e3b64f31a2b235e9f7067d52-bottom, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/81296e35c0d27ffa1ba7676ff53f87e1/cf2/testSplitTableRegionDeletedRowsDaughter=1c6b8f18205322cbb248ea4fb10155d8-ff9f5d4721ca4fc5a30cfe3141f45ca7, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/81296e35c0d27ffa1ba7676ff53f87e1/cf2/testSplitTableRegionDeletedRowsDaughter=1c6b8f18205322cbb248ea4fb10155d8-85cb745e6ba6488f8367339705326ef9, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/81296e35c0d27ffa1ba7676ff53f87e1/cf2/testSplitTableRegionDeletedRowsDaughter=1c6b8f18205322cbb248ea4fb10155d8-e6cbb96ab13a4b67a21da11e8efd233f, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/81296e35c0d27ffa1ba7676ff53f87e1/cf2/a68a5b61455b4fa0bd7b52420416f066, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/81296e35c0d27ffa1ba7676ff53f87e1/cf2/a01c4e9f61f6405380a578694839ec53.1c6b8f18205322cbb248ea4fb10155d8->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/cf2/a01c4e9f61f6405380a578694839ec53-bottom] to archive 2023-04-12 02:54:28,285 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,,1681268065445.81296e35c0d27ffa1ba7676ff53f87e1.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-12 02:54:28,287 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,,1681268065445.81296e35c0d27ffa1ba7676ff53f87e1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/81296e35c0d27ffa1ba7676ff53f87e1/cf2/926ca93402364365a7ac53b2e60b9bf5.1c6b8f18205322cbb248ea4fb10155d8 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegionDeletedRowsDaughter/81296e35c0d27ffa1ba7676ff53f87e1/cf2/926ca93402364365a7ac53b2e60b9bf5.1c6b8f18205322cbb248ea4fb10155d8 2023-04-12 02:54:28,288 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,,1681268065445.81296e35c0d27ffa1ba7676ff53f87e1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/81296e35c0d27ffa1ba7676ff53f87e1/cf2/77b0532593204cbd90904f2bbec317e7.1c6b8f18205322cbb248ea4fb10155d8 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegionDeletedRowsDaughter/81296e35c0d27ffa1ba7676ff53f87e1/cf2/77b0532593204cbd90904f2bbec317e7.1c6b8f18205322cbb248ea4fb10155d8 2023-04-12 02:54:28,292 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,,1681268065445.81296e35c0d27ffa1ba7676ff53f87e1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/81296e35c0d27ffa1ba7676ff53f87e1/cf2/381b6116e3b64f31a2b235e9f7067d52.1c6b8f18205322cbb248ea4fb10155d8 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegionDeletedRowsDaughter/81296e35c0d27ffa1ba7676ff53f87e1/cf2/381b6116e3b64f31a2b235e9f7067d52.1c6b8f18205322cbb248ea4fb10155d8 2023-04-12 02:54:28,294 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,,1681268065445.81296e35c0d27ffa1ba7676ff53f87e1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/81296e35c0d27ffa1ba7676ff53f87e1/cf2/testSplitTableRegionDeletedRowsDaughter=1c6b8f18205322cbb248ea4fb10155d8-ff9f5d4721ca4fc5a30cfe3141f45ca7 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegionDeletedRowsDaughter/81296e35c0d27ffa1ba7676ff53f87e1/cf2/testSplitTableRegionDeletedRowsDaughter=1c6b8f18205322cbb248ea4fb10155d8-ff9f5d4721ca4fc5a30cfe3141f45ca7 2023-04-12 02:54:28,295 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,,1681268065445.81296e35c0d27ffa1ba7676ff53f87e1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/81296e35c0d27ffa1ba7676ff53f87e1/cf2/testSplitTableRegionDeletedRowsDaughter=1c6b8f18205322cbb248ea4fb10155d8-85cb745e6ba6488f8367339705326ef9 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegionDeletedRowsDaughter/81296e35c0d27ffa1ba7676ff53f87e1/cf2/testSplitTableRegionDeletedRowsDaughter=1c6b8f18205322cbb248ea4fb10155d8-85cb745e6ba6488f8367339705326ef9 2023-04-12 02:54:28,297 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,,1681268065445.81296e35c0d27ffa1ba7676ff53f87e1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/81296e35c0d27ffa1ba7676ff53f87e1/cf2/testSplitTableRegionDeletedRowsDaughter=1c6b8f18205322cbb248ea4fb10155d8-e6cbb96ab13a4b67a21da11e8efd233f to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegionDeletedRowsDaughter/81296e35c0d27ffa1ba7676ff53f87e1/cf2/testSplitTableRegionDeletedRowsDaughter=1c6b8f18205322cbb248ea4fb10155d8-e6cbb96ab13a4b67a21da11e8efd233f 2023-04-12 02:54:28,299 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,,1681268065445.81296e35c0d27ffa1ba7676ff53f87e1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/81296e35c0d27ffa1ba7676ff53f87e1/cf2/a68a5b61455b4fa0bd7b52420416f066 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegionDeletedRowsDaughter/81296e35c0d27ffa1ba7676ff53f87e1/cf2/a68a5b61455b4fa0bd7b52420416f066 2023-04-12 02:54:28,301 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,,1681268065445.81296e35c0d27ffa1ba7676ff53f87e1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/81296e35c0d27ffa1ba7676ff53f87e1/cf2/a01c4e9f61f6405380a578694839ec53.1c6b8f18205322cbb248ea4fb10155d8 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegionDeletedRowsDaughter/81296e35c0d27ffa1ba7676ff53f87e1/cf2/a01c4e9f61f6405380a578694839ec53.1c6b8f18205322cbb248ea4fb10155d8 2023-04-12 02:54:28,306 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/81296e35c0d27ffa1ba7676ff53f87e1/recovered.edits/113.seqid, newMaxSeqId=113, maxSeqId=106 2023-04-12 02:54:28,307 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro 2023-04-12 02:54:28,307 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed testSplitTableRegionDeletedRowsDaughter,,1681268065445.81296e35c0d27ffa1ba7676ff53f87e1. 2023-04-12 02:54:28,307 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 81296e35c0d27ffa1ba7676ff53f87e1: 2023-04-12 02:54:28,310 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=80 updating hbase:meta row=81296e35c0d27ffa1ba7676ff53f87e1, regionState=CLOSED 2023-04-12 02:54:28,310 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testSplitTableRegionDeletedRowsDaughter,,1681268065445.81296e35c0d27ffa1ba7676ff53f87e1.","families":{"info":[{"qualifier":"regioninfo","vlen":75,"tag":[],"timestamp":"1681268068310"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681268068310"}]},"ts":"1681268068310"} 2023-04-12 02:54:28,310 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 81296e35c0d27ffa1ba7676ff53f87e1 2023-04-12 02:54:28,310 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 72b78dfd0678a920ddc3700612ff3932 2023-04-12 02:54:28,310 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 72b78dfd0678a920ddc3700612ff3932, disabling compactions & flushes 2023-04-12 02:54:28,310 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region testSplitTableRegionDeletedRowsDaughter,60,1681268065445.72b78dfd0678a920ddc3700612ff3932. 2023-04-12 02:54:28,310 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testSplitTableRegionDeletedRowsDaughter,60,1681268065445.72b78dfd0678a920ddc3700612ff3932. 2023-04-12 02:54:28,311 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on testSplitTableRegionDeletedRowsDaughter,60,1681268065445.72b78dfd0678a920ddc3700612ff3932. after waiting 0 ms 2023-04-12 02:54:28,311 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region testSplitTableRegionDeletedRowsDaughter,60,1681268065445.72b78dfd0678a920ddc3700612ff3932. 2023-04-12 02:54:28,314 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,60,1681268065445.72b78dfd0678a920ddc3700612ff3932.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/72b78dfd0678a920ddc3700612ff3932/cf1/789e1a943f304f83be48e491529ca4ad.1c6b8f18205322cbb248ea4fb10155d8->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/cf1/789e1a943f304f83be48e491529ca4ad-top, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/72b78dfd0678a920ddc3700612ff3932/cf1/0fd593a3724149b8ad466c2cef9ba846.1c6b8f18205322cbb248ea4fb10155d8->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/cf1/0fd593a3724149b8ad466c2cef9ba846-top, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/72b78dfd0678a920ddc3700612ff3932/cf1/bce8056f97d34999b5a16bbacddae4f5.1c6b8f18205322cbb248ea4fb10155d8->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/cf1/bce8056f97d34999b5a16bbacddae4f5-top, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/72b78dfd0678a920ddc3700612ff3932/cf1/fdff24c7084643c5a7298e4db0f23442.1c6b8f18205322cbb248ea4fb10155d8->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/cf1/fdff24c7084643c5a7298e4db0f23442-top, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/72b78dfd0678a920ddc3700612ff3932/cf1/testSplitTableRegionDeletedRowsDaughter=1c6b8f18205322cbb248ea4fb10155d8-4868a2af61a94040a8fcc176f05bb0fc, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/72b78dfd0678a920ddc3700612ff3932/cf1/testSplitTableRegionDeletedRowsDaughter=1c6b8f18205322cbb248ea4fb10155d8-85930b8d05664956906c5fa532cccb71, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/72b78dfd0678a920ddc3700612ff3932/cf1/f5d7697e71da4fea98fd0205d2d3a52b, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/72b78dfd0678a920ddc3700612ff3932/cf1/testSplitTableRegionDeletedRowsDaughter=1c6b8f18205322cbb248ea4fb10155d8-45831fa527a845c783019394c67c0fa7] to archive 2023-04-12 02:54:28,315 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,60,1681268065445.72b78dfd0678a920ddc3700612ff3932.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-12 02:54:28,315 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=83, resume processing ppid=80 2023-04-12 02:54:28,315 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=83, ppid=80, state=SUCCESS; CloseRegionProcedure 81296e35c0d27ffa1ba7676ff53f87e1, server=jenkins-hbase4.apache.org,34579,1681268031719 in 189 msec 2023-04-12 02:54:28,318 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=80, ppid=79, state=SUCCESS; TransitRegionStateProcedure table=testSplitTableRegionDeletedRowsDaughter, region=81296e35c0d27ffa1ba7676ff53f87e1, UNASSIGN in 204 msec 2023-04-12 02:54:28,318 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,60,1681268065445.72b78dfd0678a920ddc3700612ff3932.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/72b78dfd0678a920ddc3700612ff3932/cf1/789e1a943f304f83be48e491529ca4ad.1c6b8f18205322cbb248ea4fb10155d8 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegionDeletedRowsDaughter/72b78dfd0678a920ddc3700612ff3932/cf1/789e1a943f304f83be48e491529ca4ad.1c6b8f18205322cbb248ea4fb10155d8 2023-04-12 02:54:28,319 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,60,1681268065445.72b78dfd0678a920ddc3700612ff3932.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/72b78dfd0678a920ddc3700612ff3932/cf1/0fd593a3724149b8ad466c2cef9ba846.1c6b8f18205322cbb248ea4fb10155d8 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegionDeletedRowsDaughter/72b78dfd0678a920ddc3700612ff3932/cf1/0fd593a3724149b8ad466c2cef9ba846.1c6b8f18205322cbb248ea4fb10155d8 2023-04-12 02:54:28,321 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,60,1681268065445.72b78dfd0678a920ddc3700612ff3932.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/72b78dfd0678a920ddc3700612ff3932/cf1/bce8056f97d34999b5a16bbacddae4f5.1c6b8f18205322cbb248ea4fb10155d8 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegionDeletedRowsDaughter/72b78dfd0678a920ddc3700612ff3932/cf1/bce8056f97d34999b5a16bbacddae4f5.1c6b8f18205322cbb248ea4fb10155d8 2023-04-12 02:54:28,323 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,60,1681268065445.72b78dfd0678a920ddc3700612ff3932.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/72b78dfd0678a920ddc3700612ff3932/cf1/fdff24c7084643c5a7298e4db0f23442.1c6b8f18205322cbb248ea4fb10155d8 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegionDeletedRowsDaughter/72b78dfd0678a920ddc3700612ff3932/cf1/fdff24c7084643c5a7298e4db0f23442.1c6b8f18205322cbb248ea4fb10155d8 2023-04-12 02:54:28,325 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,60,1681268065445.72b78dfd0678a920ddc3700612ff3932.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/72b78dfd0678a920ddc3700612ff3932/cf1/testSplitTableRegionDeletedRowsDaughter=1c6b8f18205322cbb248ea4fb10155d8-4868a2af61a94040a8fcc176f05bb0fc to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegionDeletedRowsDaughter/72b78dfd0678a920ddc3700612ff3932/cf1/testSplitTableRegionDeletedRowsDaughter=1c6b8f18205322cbb248ea4fb10155d8-4868a2af61a94040a8fcc176f05bb0fc 2023-04-12 02:54:28,327 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,60,1681268065445.72b78dfd0678a920ddc3700612ff3932.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/72b78dfd0678a920ddc3700612ff3932/cf1/testSplitTableRegionDeletedRowsDaughter=1c6b8f18205322cbb248ea4fb10155d8-85930b8d05664956906c5fa532cccb71 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegionDeletedRowsDaughter/72b78dfd0678a920ddc3700612ff3932/cf1/testSplitTableRegionDeletedRowsDaughter=1c6b8f18205322cbb248ea4fb10155d8-85930b8d05664956906c5fa532cccb71 2023-04-12 02:54:28,329 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,60,1681268065445.72b78dfd0678a920ddc3700612ff3932.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/72b78dfd0678a920ddc3700612ff3932/cf1/f5d7697e71da4fea98fd0205d2d3a52b to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegionDeletedRowsDaughter/72b78dfd0678a920ddc3700612ff3932/cf1/f5d7697e71da4fea98fd0205d2d3a52b 2023-04-12 02:54:28,333 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,60,1681268065445.72b78dfd0678a920ddc3700612ff3932.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/72b78dfd0678a920ddc3700612ff3932/cf1/testSplitTableRegionDeletedRowsDaughter=1c6b8f18205322cbb248ea4fb10155d8-45831fa527a845c783019394c67c0fa7 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegionDeletedRowsDaughter/72b78dfd0678a920ddc3700612ff3932/cf1/testSplitTableRegionDeletedRowsDaughter=1c6b8f18205322cbb248ea4fb10155d8-45831fa527a845c783019394c67c0fa7 2023-04-12 02:54:28,337 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,60,1681268065445.72b78dfd0678a920ddc3700612ff3932.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/72b78dfd0678a920ddc3700612ff3932/cf2/926ca93402364365a7ac53b2e60b9bf5.1c6b8f18205322cbb248ea4fb10155d8->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/cf2/926ca93402364365a7ac53b2e60b9bf5-top, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/72b78dfd0678a920ddc3700612ff3932/cf2/77b0532593204cbd90904f2bbec317e7.1c6b8f18205322cbb248ea4fb10155d8->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/cf2/77b0532593204cbd90904f2bbec317e7-top, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/72b78dfd0678a920ddc3700612ff3932/cf2/381b6116e3b64f31a2b235e9f7067d52.1c6b8f18205322cbb248ea4fb10155d8->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/cf2/381b6116e3b64f31a2b235e9f7067d52-top, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/72b78dfd0678a920ddc3700612ff3932/cf2/a01c4e9f61f6405380a578694839ec53.1c6b8f18205322cbb248ea4fb10155d8->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/cf2/a01c4e9f61f6405380a578694839ec53-top, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/72b78dfd0678a920ddc3700612ff3932/cf2/testSplitTableRegionDeletedRowsDaughter=1c6b8f18205322cbb248ea4fb10155d8-0c6c01c95a4f4bc3a4312a2eee9e8eec, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/72b78dfd0678a920ddc3700612ff3932/cf2/testSplitTableRegionDeletedRowsDaughter=1c6b8f18205322cbb248ea4fb10155d8-37083e44dea546c8a8710fe5148a6dcd, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/72b78dfd0678a920ddc3700612ff3932/cf2/96fe5b19e4694f46927e13df8f2c6a66, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/72b78dfd0678a920ddc3700612ff3932/cf2/testSplitTableRegionDeletedRowsDaughter=1c6b8f18205322cbb248ea4fb10155d8-8ac99bc8ac8f4c3b873212471489db78] to archive 2023-04-12 02:54:28,338 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,60,1681268065445.72b78dfd0678a920ddc3700612ff3932.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-12 02:54:28,340 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,60,1681268065445.72b78dfd0678a920ddc3700612ff3932.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/72b78dfd0678a920ddc3700612ff3932/cf2/926ca93402364365a7ac53b2e60b9bf5.1c6b8f18205322cbb248ea4fb10155d8 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegionDeletedRowsDaughter/72b78dfd0678a920ddc3700612ff3932/cf2/926ca93402364365a7ac53b2e60b9bf5.1c6b8f18205322cbb248ea4fb10155d8 2023-04-12 02:54:28,342 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,60,1681268065445.72b78dfd0678a920ddc3700612ff3932.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/72b78dfd0678a920ddc3700612ff3932/cf2/77b0532593204cbd90904f2bbec317e7.1c6b8f18205322cbb248ea4fb10155d8 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegionDeletedRowsDaughter/72b78dfd0678a920ddc3700612ff3932/cf2/77b0532593204cbd90904f2bbec317e7.1c6b8f18205322cbb248ea4fb10155d8 2023-04-12 02:54:28,343 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,60,1681268065445.72b78dfd0678a920ddc3700612ff3932.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/72b78dfd0678a920ddc3700612ff3932/cf2/381b6116e3b64f31a2b235e9f7067d52.1c6b8f18205322cbb248ea4fb10155d8 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegionDeletedRowsDaughter/72b78dfd0678a920ddc3700612ff3932/cf2/381b6116e3b64f31a2b235e9f7067d52.1c6b8f18205322cbb248ea4fb10155d8 2023-04-12 02:54:28,345 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,60,1681268065445.72b78dfd0678a920ddc3700612ff3932.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/72b78dfd0678a920ddc3700612ff3932/cf2/a01c4e9f61f6405380a578694839ec53.1c6b8f18205322cbb248ea4fb10155d8 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegionDeletedRowsDaughter/72b78dfd0678a920ddc3700612ff3932/cf2/a01c4e9f61f6405380a578694839ec53.1c6b8f18205322cbb248ea4fb10155d8 2023-04-12 02:54:28,346 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,60,1681268065445.72b78dfd0678a920ddc3700612ff3932.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/72b78dfd0678a920ddc3700612ff3932/cf2/testSplitTableRegionDeletedRowsDaughter=1c6b8f18205322cbb248ea4fb10155d8-0c6c01c95a4f4bc3a4312a2eee9e8eec to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegionDeletedRowsDaughter/72b78dfd0678a920ddc3700612ff3932/cf2/testSplitTableRegionDeletedRowsDaughter=1c6b8f18205322cbb248ea4fb10155d8-0c6c01c95a4f4bc3a4312a2eee9e8eec 2023-04-12 02:54:28,348 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,60,1681268065445.72b78dfd0678a920ddc3700612ff3932.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/72b78dfd0678a920ddc3700612ff3932/cf2/testSplitTableRegionDeletedRowsDaughter=1c6b8f18205322cbb248ea4fb10155d8-37083e44dea546c8a8710fe5148a6dcd to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegionDeletedRowsDaughter/72b78dfd0678a920ddc3700612ff3932/cf2/testSplitTableRegionDeletedRowsDaughter=1c6b8f18205322cbb248ea4fb10155d8-37083e44dea546c8a8710fe5148a6dcd 2023-04-12 02:54:28,349 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,60,1681268065445.72b78dfd0678a920ddc3700612ff3932.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/72b78dfd0678a920ddc3700612ff3932/cf2/96fe5b19e4694f46927e13df8f2c6a66 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegionDeletedRowsDaughter/72b78dfd0678a920ddc3700612ff3932/cf2/96fe5b19e4694f46927e13df8f2c6a66 2023-04-12 02:54:28,350 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,60,1681268065445.72b78dfd0678a920ddc3700612ff3932.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/72b78dfd0678a920ddc3700612ff3932/cf2/testSplitTableRegionDeletedRowsDaughter=1c6b8f18205322cbb248ea4fb10155d8-8ac99bc8ac8f4c3b873212471489db78 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegionDeletedRowsDaughter/72b78dfd0678a920ddc3700612ff3932/cf2/testSplitTableRegionDeletedRowsDaughter=1c6b8f18205322cbb248ea4fb10155d8-8ac99bc8ac8f4c3b873212471489db78 2023-04-12 02:54:28,354 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionDeletedRowsDaughter/72b78dfd0678a920ddc3700612ff3932/recovered.edits/113.seqid, newMaxSeqId=113, maxSeqId=106 2023-04-12 02:54:28,355 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro 2023-04-12 02:54:28,355 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed testSplitTableRegionDeletedRowsDaughter,60,1681268065445.72b78dfd0678a920ddc3700612ff3932. 2023-04-12 02:54:28,355 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 72b78dfd0678a920ddc3700612ff3932: 2023-04-12 02:54:28,357 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 72b78dfd0678a920ddc3700612ff3932 2023-04-12 02:54:28,357 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=81 updating hbase:meta row=72b78dfd0678a920ddc3700612ff3932, regionState=CLOSED 2023-04-12 02:54:28,357 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testSplitTableRegionDeletedRowsDaughter,60,1681268065445.72b78dfd0678a920ddc3700612ff3932.","families":{"info":[{"qualifier":"regioninfo","vlen":75,"tag":[],"timestamp":"1681268068357"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681268068357"}]},"ts":"1681268068357"} 2023-04-12 02:54:28,366 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=82, resume processing ppid=81 2023-04-12 02:54:28,366 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=82, ppid=81, state=SUCCESS; CloseRegionProcedure 72b78dfd0678a920ddc3700612ff3932, server=jenkins-hbase4.apache.org,34579,1681268031719 in 248 msec 2023-04-12 02:54:28,369 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=81, resume processing ppid=79 2023-04-12 02:54:28,369 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=81, ppid=79, state=SUCCESS; TransitRegionStateProcedure table=testSplitTableRegionDeletedRowsDaughter, region=72b78dfd0678a920ddc3700612ff3932, UNASSIGN in 254 msec 2023-04-12 02:54:28,370 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testSplitTableRegionDeletedRowsDaughter","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681268068370"}]},"ts":"1681268068370"} 2023-04-12 02:54:28,371 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testSplitTableRegionDeletedRowsDaughter, state=DISABLED in hbase:meta 2023-04-12 02:54:28,373 INFO [PEWorker-1] procedure.DisableTableProcedure(305): Set testSplitTableRegionDeletedRowsDaughter to state=DISABLED 2023-04-12 02:54:28,374 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=79, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionDeletedRowsDaughter in 277 msec 2023-04-12 02:54:28,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] master.MasterRpcServices(1227): Checking to see if procedure is done pid=79 2023-04-12 02:54:28,413 INFO [Listener at localhost/46667] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testSplitTableRegionDeletedRowsDaughter, procId: 79 completed 2023-04-12 02:54:28,414 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:28,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] procedure2.ProcedureExecutor(1029): Stored pid=84, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:28,417 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=84, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:28,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] master.MasterRpcServices(1227): Checking to see if procedure is done pid=84 2023-04-12 02:54:28,420 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=84, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:28,427 DEBUG [HFileArchiver-8] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8 2023-04-12 02:54:28,427 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/81296e35c0d27ffa1ba7676ff53f87e1 2023-04-12 02:54:28,427 DEBUG [HFileArchiver-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/72b78dfd0678a920ddc3700612ff3932 2023-04-12 02:54:28,430 DEBUG [HFileArchiver-5] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/72b78dfd0678a920ddc3700612ff3932/cf1, FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/72b78dfd0678a920ddc3700612ff3932/cf2, FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/72b78dfd0678a920ddc3700612ff3932/recovered.edits] 2023-04-12 02:54:28,432 DEBUG [HFileArchiver-8] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/cf1, FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/cf2, FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/recovered.edits] 2023-04-12 02:54:28,432 DEBUG [HFileArchiver-2] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/81296e35c0d27ffa1ba7676ff53f87e1/cf1, FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/81296e35c0d27ffa1ba7676ff53f87e1/cf2, FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/81296e35c0d27ffa1ba7676ff53f87e1/recovered.edits] 2023-04-12 02:54:28,435 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/72b78dfd0678a920ddc3700612ff3932/cf1/95b6df5c670c47fdbb882ac67b11ad32 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegionDeletedRowsDaughter/72b78dfd0678a920ddc3700612ff3932/cf1/95b6df5c670c47fdbb882ac67b11ad32 2023-04-12 02:54:28,435 DEBUG [HFileArchiver-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/81296e35c0d27ffa1ba7676ff53f87e1/cf1/9739f85c05884e7bbee8b2cf14b73c87 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegionDeletedRowsDaughter/81296e35c0d27ffa1ba7676ff53f87e1/cf1/9739f85c05884e7bbee8b2cf14b73c87 2023-04-12 02:54:28,437 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/cf1/0fd593a3724149b8ad466c2cef9ba846 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/cf1/0fd593a3724149b8ad466c2cef9ba846 2023-04-12 02:54:28,438 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/72b78dfd0678a920ddc3700612ff3932/cf2/069237fd2acc4ec79a042514c86be2bf to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegionDeletedRowsDaughter/72b78dfd0678a920ddc3700612ff3932/cf2/069237fd2acc4ec79a042514c86be2bf 2023-04-12 02:54:28,438 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/cf1/45831fa527a845c783019394c67c0fa7 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/cf1/45831fa527a845c783019394c67c0fa7 2023-04-12 02:54:28,440 DEBUG [HFileArchiver-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/81296e35c0d27ffa1ba7676ff53f87e1/cf2/4a5dbb079f0845959e0cd2c5ba7122bf to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegionDeletedRowsDaughter/81296e35c0d27ffa1ba7676ff53f87e1/cf2/4a5dbb079f0845959e0cd2c5ba7122bf 2023-04-12 02:54:28,441 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/cf1/4868a2af61a94040a8fcc176f05bb0fc to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/cf1/4868a2af61a94040a8fcc176f05bb0fc 2023-04-12 02:54:28,451 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/cf1/710424cef2c64d2ebc740c81294a95cb to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/cf1/710424cef2c64d2ebc740c81294a95cb 2023-04-12 02:54:28,453 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/72b78dfd0678a920ddc3700612ff3932/recovered.edits/113.seqid to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegionDeletedRowsDaughter/72b78dfd0678a920ddc3700612ff3932/recovered.edits/113.seqid 2023-04-12 02:54:28,453 DEBUG [HFileArchiver-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/81296e35c0d27ffa1ba7676ff53f87e1/recovered.edits/113.seqid to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegionDeletedRowsDaughter/81296e35c0d27ffa1ba7676ff53f87e1/recovered.edits/113.seqid 2023-04-12 02:54:28,454 DEBUG [HFileArchiver-2] backup.HFileArchiver(596): Deleted hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/81296e35c0d27ffa1ba7676ff53f87e1 2023-04-12 02:54:28,454 DEBUG [HFileArchiver-5] backup.HFileArchiver(596): Deleted hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/72b78dfd0678a920ddc3700612ff3932 2023-04-12 02:54:28,454 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/cf1/789e1a943f304f83be48e491529ca4ad to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/cf1/789e1a943f304f83be48e491529ca4ad 2023-04-12 02:54:28,456 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/cf1/85930b8d05664956906c5fa532cccb71 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/cf1/85930b8d05664956906c5fa532cccb71 2023-04-12 02:54:28,460 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/cf1/985388b30d784536a1f270feb7f52366 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/cf1/985388b30d784536a1f270feb7f52366 2023-04-12 02:54:28,463 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/cf1/9c4efc02782d420291d1fe480cade0d3 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/cf1/9c4efc02782d420291d1fe480cade0d3 2023-04-12 02:54:28,465 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/cf1/bce8056f97d34999b5a16bbacddae4f5 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/cf1/bce8056f97d34999b5a16bbacddae4f5 2023-04-12 02:54:28,467 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/cf1/fdff24c7084643c5a7298e4db0f23442 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/cf1/fdff24c7084643c5a7298e4db0f23442 2023-04-12 02:54:28,472 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/cf2/0c6c01c95a4f4bc3a4312a2eee9e8eec to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/cf2/0c6c01c95a4f4bc3a4312a2eee9e8eec 2023-04-12 02:54:28,475 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/cf2/37083e44dea546c8a8710fe5148a6dcd to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/cf2/37083e44dea546c8a8710fe5148a6dcd 2023-04-12 02:54:28,479 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/cf2/381b6116e3b64f31a2b235e9f7067d52 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/cf2/381b6116e3b64f31a2b235e9f7067d52 2023-04-12 02:54:28,481 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/cf2/77b0532593204cbd90904f2bbec317e7 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/cf2/77b0532593204cbd90904f2bbec317e7 2023-04-12 02:54:28,482 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/cf2/85cb745e6ba6488f8367339705326ef9 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/cf2/85cb745e6ba6488f8367339705326ef9 2023-04-12 02:54:28,484 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/cf2/8ac99bc8ac8f4c3b873212471489db78 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/cf2/8ac99bc8ac8f4c3b873212471489db78 2023-04-12 02:54:28,485 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/cf2/926ca93402364365a7ac53b2e60b9bf5 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/cf2/926ca93402364365a7ac53b2e60b9bf5 2023-04-12 02:54:28,486 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/cf2/a01c4e9f61f6405380a578694839ec53 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/cf2/a01c4e9f61f6405380a578694839ec53 2023-04-12 02:54:28,488 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/cf2/e6cbb96ab13a4b67a21da11e8efd233f to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/cf2/e6cbb96ab13a4b67a21da11e8efd233f 2023-04-12 02:54:28,490 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/cf2/ff9f5d4721ca4fc5a30cfe3141f45ca7 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/cf2/ff9f5d4721ca4fc5a30cfe3141f45ca7 2023-04-12 02:54:28,493 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/recovered.edits/106.seqid to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8/recovered.edits/106.seqid 2023-04-12 02:54:28,493 DEBUG [HFileArchiver-8] backup.HFileArchiver(596): Deleted hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/1c6b8f18205322cbb248ea4fb10155d8 2023-04-12 02:54:28,494 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived testSplitTableRegionDeletedRowsDaughter regions 2023-04-12 02:54:28,498 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=84, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:28,501 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 3 rows of testSplitTableRegionDeletedRowsDaughter from hbase:meta 2023-04-12 02:54:28,506 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'testSplitTableRegionDeletedRowsDaughter' descriptor. 2023-04-12 02:54:28,509 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=84, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:28,509 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'testSplitTableRegionDeletedRowsDaughter' from region states. 2023-04-12 02:54:28,510 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testSplitTableRegionDeletedRowsDaughter,,1681268062677.1c6b8f18205322cbb248ea4fb10155d8.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1681268068509"}]},"ts":"9223372036854775807"} 2023-04-12 02:54:28,510 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testSplitTableRegionDeletedRowsDaughter,,1681268065445.81296e35c0d27ffa1ba7676ff53f87e1.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1681268068509"}]},"ts":"9223372036854775807"} 2023-04-12 02:54:28,510 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testSplitTableRegionDeletedRowsDaughter,60,1681268065445.72b78dfd0678a920ddc3700612ff3932.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1681268068509"}]},"ts":"9223372036854775807"} 2023-04-12 02:54:28,512 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 3 regions from META 2023-04-12 02:54:28,512 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 1c6b8f18205322cbb248ea4fb10155d8, NAME => 'testSplitTableRegionDeletedRowsDaughter,,1681268062677.1c6b8f18205322cbb248ea4fb10155d8.', STARTKEY => '', ENDKEY => ''}, {ENCODED => 81296e35c0d27ffa1ba7676ff53f87e1, NAME => 'testSplitTableRegionDeletedRowsDaughter,,1681268065445.81296e35c0d27ffa1ba7676ff53f87e1.', STARTKEY => '', ENDKEY => '60'}, {ENCODED => 72b78dfd0678a920ddc3700612ff3932, NAME => 'testSplitTableRegionDeletedRowsDaughter,60,1681268065445.72b78dfd0678a920ddc3700612ff3932.', STARTKEY => '60', ENDKEY => ''}] 2023-04-12 02:54:28,512 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'testSplitTableRegionDeletedRowsDaughter' as deleted. 2023-04-12 02:54:28,512 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testSplitTableRegionDeletedRowsDaughter","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1681268068512"}]},"ts":"9223372036854775807"} 2023-04-12 02:54:28,514 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table testSplitTableRegionDeletedRowsDaughter state from META 2023-04-12 02:54:28,516 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=84, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:54:28,518 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=84, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionDeletedRowsDaughter in 102 msec 2023-04-12 02:54:28,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] master.MasterRpcServices(1227): Checking to see if procedure is done pid=84 2023-04-12 02:54:28,519 INFO [Listener at localhost/46667] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:testSplitTableRegionDeletedRowsDaughter, procId: 84 completed 2023-04-12 02:54:28,534 INFO [Listener at localhost/46667] hbase.ResourceChecker(175): after: master.assignment.TestSplitTableRegionProcedure#testSplitTableRegionDeletedRowsDaughter Thread=408 (was 402) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1018420944_15 at /127.0.0.1:37570 [Waiting for operation #3] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1018420944_15 at /127.0.0.1:37554 [Waiting for operation #2] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1018420944_15 at /127.0.0.1:37538 [Waiting for operation #5] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1018420944_15 at /127.0.0.1:54100 [Waiting for operation #9] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x2d8012cd-metaLookup-shared--pool-13 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x6cb27db9-shared-pool-19 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x6cb27db9-shared-pool-23 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1018420944_15 at /127.0.0.1:60372 [Waiting for operation #4] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x2d8012cd-shared-pool-14 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x6cb27db9-shared-pool-22 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1018420944_15 at /127.0.0.1:60354 [Waiting for operation #2] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x2d8012cd-metaLookup-shared--pool-12 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1018420944_15 at /127.0.0.1:60366 [Waiting for operation #3] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1018420944_15 at /127.0.0.1:37586 [Waiting for operation #5] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x2d8012cd-shared-pool-15 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x2d8012cd-shared-pool-16 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1018420944_15 at /127.0.0.1:60360 [Waiting for operation #6] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x2d8012cd-shared-pool-17 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x2d8012cd-metaLookup-shared--pool-11 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x6cb27db9-shared-pool-20 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1018420944_15 at /127.0.0.1:57526 [Waiting for operation #17] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x6cb27db9-shared-pool-21 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x2d8012cd-shared-pool-13 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x2d8012cd-shared-pool-18 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-4 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:883) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1018420944_15 at /127.0.0.1:60050 [Waiting for operation #21] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x2d8012cd-metaLookup-shared--pool-14 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1018420944_15 at /127.0.0.1:54178 [Waiting for operation #3] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1018420944_15 at /127.0.0.1:60340 [Waiting for operation #3] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1018420944_15 at /127.0.0.1:60382 [Waiting for operation #2] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x2d8012cd-metaLookup-shared--pool-10 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1018420944_15 at /127.0.0.1:60384 [Waiting for operation #5] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1018420944_15 at /127.0.0.1:37518 [Waiting for operation #5] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) - Thread LEAK? -, OpenFileDescriptor=835 (was 832) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=423 (was 425), ProcessCount=172 (was 172), AvailableMemoryMB=1038 (was 1130) 2023-04-12 02:54:28,551 INFO [Listener at localhost/46667] hbase.ResourceChecker(147): before: master.assignment.TestSplitTableRegionProcedure#testRollbackForSplitTableRegionWithReplica Thread=408, OpenFileDescriptor=835, MaxFileDescriptor=60000, SystemLoadAverage=423, ProcessCount=172, AvailableMemoryMB=1038 2023-04-12 02:54:28,551 WARN [Listener at localhost/46667] procedure2.ProcedureTestingUtility(193): Set Kill before store update to: false 2023-04-12 02:54:28,554 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/balancer 2023-04-12 02:54:28,554 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] master.MasterRpcServices(492): Client=jenkins//172.31.14.131 set balanceSwitch=false 2023-04-12 02:54:28,555 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(1029): Stored pid=85, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-04-12 02:54:28,559 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=85, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=testRollbackForSplitTableRegionWithReplica execute state=CREATE_TABLE_PRE_OPERATION 2023-04-12 02:54:28,560 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=85, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=testRollbackForSplitTableRegionWithReplica execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-12 02:54:28,562 DEBUG [HFileArchiver-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testRollbackForSplitTableRegionWithReplica/4381c9c8e57c5dbb2533908abaf07cf1 2023-04-12 02:54:28,562 DEBUG [HFileArchiver-4] backup.HFileArchiver(153): Directory hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testRollbackForSplitTableRegionWithReplica/4381c9c8e57c5dbb2533908abaf07cf1 empty. 2023-04-12 02:54:28,563 DEBUG [HFileArchiver-4] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testRollbackForSplitTableRegionWithReplica/4381c9c8e57c5dbb2533908abaf07cf1 2023-04-12 02:54:28,563 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived testRollbackForSplitTableRegionWithReplica regions 2023-04-12 02:54:28,580 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testRollbackForSplitTableRegionWithReplica/.tabledesc/.tableinfo.0000000001 2023-04-12 02:54:28,581 INFO [RegionOpenAndInit-testRollbackForSplitTableRegionWithReplica-pool-0] regionserver.HRegion(7675): creating {ENCODED => 4381c9c8e57c5dbb2533908abaf07cf1, NAME => 'testRollbackForSplitTableRegionWithReplica,,1681268068555.4381c9c8e57c5dbb2533908abaf07cf1.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='testRollbackForSplitTableRegionWithReplica', {NAME => 'cf1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp 2023-04-12 02:54:28,614 DEBUG [RegionOpenAndInit-testRollbackForSplitTableRegionWithReplica-pool-0] regionserver.HRegion(866): Instantiated testRollbackForSplitTableRegionWithReplica,,1681268068555.4381c9c8e57c5dbb2533908abaf07cf1.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-12 02:54:28,614 DEBUG [RegionOpenAndInit-testRollbackForSplitTableRegionWithReplica-pool-0] regionserver.HRegion(1604): Closing 4381c9c8e57c5dbb2533908abaf07cf1, disabling compactions & flushes 2023-04-12 02:54:28,614 INFO [RegionOpenAndInit-testRollbackForSplitTableRegionWithReplica-pool-0] regionserver.HRegion(1626): Closing region testRollbackForSplitTableRegionWithReplica,,1681268068555.4381c9c8e57c5dbb2533908abaf07cf1. 2023-04-12 02:54:28,615 DEBUG [RegionOpenAndInit-testRollbackForSplitTableRegionWithReplica-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testRollbackForSplitTableRegionWithReplica,,1681268068555.4381c9c8e57c5dbb2533908abaf07cf1. 2023-04-12 02:54:28,615 DEBUG [RegionOpenAndInit-testRollbackForSplitTableRegionWithReplica-pool-0] regionserver.HRegion(1714): Acquired close lock on testRollbackForSplitTableRegionWithReplica,,1681268068555.4381c9c8e57c5dbb2533908abaf07cf1. after waiting 0 ms 2023-04-12 02:54:28,615 DEBUG [RegionOpenAndInit-testRollbackForSplitTableRegionWithReplica-pool-0] regionserver.HRegion(1724): Updates disabled for region testRollbackForSplitTableRegionWithReplica,,1681268068555.4381c9c8e57c5dbb2533908abaf07cf1. 2023-04-12 02:54:28,615 INFO [RegionOpenAndInit-testRollbackForSplitTableRegionWithReplica-pool-0] regionserver.HRegion(1838): Closed testRollbackForSplitTableRegionWithReplica,,1681268068555.4381c9c8e57c5dbb2533908abaf07cf1. 2023-04-12 02:54:28,615 DEBUG [RegionOpenAndInit-testRollbackForSplitTableRegionWithReplica-pool-0] regionserver.HRegion(1558): Region close journal for 4381c9c8e57c5dbb2533908abaf07cf1: 2023-04-12 02:54:28,617 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=85, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=testRollbackForSplitTableRegionWithReplica execute state=CREATE_TABLE_ADD_TO_META 2023-04-12 02:54:28,618 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRollbackForSplitTableRegionWithReplica,,1681268068555.4381c9c8e57c5dbb2533908abaf07cf1.","families":{"info":[{"qualifier":"regioninfo","vlen":76,"tag":[],"timestamp":"1681268068618"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681268068618"}]},"ts":"1681268068618"} 2023-04-12 02:54:28,620 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-12 02:54:28,621 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=85, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=testRollbackForSplitTableRegionWithReplica execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-12 02:54:28,621 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRollbackForSplitTableRegionWithReplica","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681268068621"}]},"ts":"1681268068621"} 2023-04-12 02:54:28,622 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testRollbackForSplitTableRegionWithReplica, state=ENABLING in hbase:meta 2023-04-12 02:54:28,625 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-04-12 02:54:28,626 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-04-12 02:54:28,626 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-04-12 02:54:28,626 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-04-12 02:54:28,626 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-04-12 02:54:28,626 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=86, ppid=85, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRollbackForSplitTableRegionWithReplica, region=4381c9c8e57c5dbb2533908abaf07cf1, ASSIGN}] 2023-04-12 02:54:28,631 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=86, ppid=85, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRollbackForSplitTableRegionWithReplica, region=4381c9c8e57c5dbb2533908abaf07cf1, ASSIGN 2023-04-12 02:54:28,632 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=86, ppid=85, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRollbackForSplitTableRegionWithReplica, region=4381c9c8e57c5dbb2533908abaf07cf1, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,43091,1681268031804; forceNewPlan=false, retain=false 2023-04-12 02:54:28,782 INFO [jenkins-hbase4:40281] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-04-12 02:54:28,785 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=86 updating hbase:meta row=4381c9c8e57c5dbb2533908abaf07cf1, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:28,785 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRollbackForSplitTableRegionWithReplica,,1681268068555.4381c9c8e57c5dbb2533908abaf07cf1.","families":{"info":[{"qualifier":"regioninfo","vlen":76,"tag":[],"timestamp":"1681268068785"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681268068785"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681268068785"}]},"ts":"1681268068785"} 2023-04-12 02:54:28,787 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=87, ppid=86, state=RUNNABLE; OpenRegionProcedure 4381c9c8e57c5dbb2533908abaf07cf1, server=jenkins-hbase4.apache.org,43091,1681268031804}] 2023-04-12 02:54:28,939 DEBUG [RSProcedureDispatcher-pool-1] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:28,939 DEBUG [RSProcedureDispatcher-pool-1] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-04-12 02:54:28,943 INFO [RS-EventLoopGroup-5-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:38332, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-04-12 02:54:28,947 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRollbackForSplitTableRegionWithReplica,,1681268068555.4381c9c8e57c5dbb2533908abaf07cf1. 2023-04-12 02:54:28,947 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 4381c9c8e57c5dbb2533908abaf07cf1, NAME => 'testRollbackForSplitTableRegionWithReplica,,1681268068555.4381c9c8e57c5dbb2533908abaf07cf1.', STARTKEY => '', ENDKEY => ''} 2023-04-12 02:54:28,947 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(174): System coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro loaded, priority=536870911. 2023-04-12 02:54:28,948 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRollbackForSplitTableRegionWithReplica 4381c9c8e57c5dbb2533908abaf07cf1 2023-04-12 02:54:28,948 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated testRollbackForSplitTableRegionWithReplica,,1681268068555.4381c9c8e57c5dbb2533908abaf07cf1.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-12 02:54:28,948 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 4381c9c8e57c5dbb2533908abaf07cf1 2023-04-12 02:54:28,948 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 4381c9c8e57c5dbb2533908abaf07cf1 2023-04-12 02:54:28,949 INFO [StoreOpener-4381c9c8e57c5dbb2533908abaf07cf1-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf1 of region 4381c9c8e57c5dbb2533908abaf07cf1 2023-04-12 02:54:28,951 DEBUG [StoreOpener-4381c9c8e57c5dbb2533908abaf07cf1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackForSplitTableRegionWithReplica/4381c9c8e57c5dbb2533908abaf07cf1/cf1 2023-04-12 02:54:28,951 DEBUG [StoreOpener-4381c9c8e57c5dbb2533908abaf07cf1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackForSplitTableRegionWithReplica/4381c9c8e57c5dbb2533908abaf07cf1/cf1 2023-04-12 02:54:28,951 INFO [StoreOpener-4381c9c8e57c5dbb2533908abaf07cf1-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 4381c9c8e57c5dbb2533908abaf07cf1 columnFamilyName cf1 2023-04-12 02:54:28,951 INFO [StoreOpener-4381c9c8e57c5dbb2533908abaf07cf1-1] regionserver.HStore(310): Store=4381c9c8e57c5dbb2533908abaf07cf1/cf1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-12 02:54:28,952 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackForSplitTableRegionWithReplica/4381c9c8e57c5dbb2533908abaf07cf1 2023-04-12 02:54:28,953 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackForSplitTableRegionWithReplica/4381c9c8e57c5dbb2533908abaf07cf1 2023-04-12 02:54:28,955 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 4381c9c8e57c5dbb2533908abaf07cf1 2023-04-12 02:54:28,957 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackForSplitTableRegionWithReplica/4381c9c8e57c5dbb2533908abaf07cf1/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-12 02:54:28,958 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 4381c9c8e57c5dbb2533908abaf07cf1; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=66160480, jitterRate=-0.014132022857666016}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-04-12 02:54:28,958 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 4381c9c8e57c5dbb2533908abaf07cf1: 2023-04-12 02:54:28,959 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRollbackForSplitTableRegionWithReplica,,1681268068555.4381c9c8e57c5dbb2533908abaf07cf1., pid=87, masterSystemTime=1681268068939 2023-04-12 02:54:28,962 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRollbackForSplitTableRegionWithReplica,,1681268068555.4381c9c8e57c5dbb2533908abaf07cf1. 2023-04-12 02:54:28,963 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRollbackForSplitTableRegionWithReplica,,1681268068555.4381c9c8e57c5dbb2533908abaf07cf1. 2023-04-12 02:54:28,963 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=86 updating hbase:meta row=4381c9c8e57c5dbb2533908abaf07cf1, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:28,963 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRollbackForSplitTableRegionWithReplica,,1681268068555.4381c9c8e57c5dbb2533908abaf07cf1.","families":{"info":[{"qualifier":"regioninfo","vlen":76,"tag":[],"timestamp":"1681268068963"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1681268068963"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681268068963"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681268068963"}]},"ts":"1681268068963"} 2023-04-12 02:54:28,967 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=87, resume processing ppid=86 2023-04-12 02:54:28,967 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=87, ppid=86, state=SUCCESS; OpenRegionProcedure 4381c9c8e57c5dbb2533908abaf07cf1, server=jenkins-hbase4.apache.org,43091,1681268031804 in 178 msec 2023-04-12 02:54:28,973 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=86, resume processing ppid=85 2023-04-12 02:54:28,973 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=86, ppid=85, state=SUCCESS; TransitRegionStateProcedure table=testRollbackForSplitTableRegionWithReplica, region=4381c9c8e57c5dbb2533908abaf07cf1, ASSIGN in 341 msec 2023-04-12 02:54:28,974 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=85, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=testRollbackForSplitTableRegionWithReplica execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-12 02:54:28,974 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRollbackForSplitTableRegionWithReplica","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681268068974"}]},"ts":"1681268068974"} 2023-04-12 02:54:28,975 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testRollbackForSplitTableRegionWithReplica, state=ENABLED in hbase:meta 2023-04-12 02:54:28,981 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=85, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=testRollbackForSplitTableRegionWithReplica execute state=CREATE_TABLE_POST_OPERATION 2023-04-12 02:54:28,982 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=85, state=SUCCESS; CreateTableProcedure table=testRollbackForSplitTableRegionWithReplica in 426 msec 2023-04-12 02:54:29,066 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] master.HMaster$12(2454): Client=jenkins//172.31.14.131 modify table testRollbackForSplitTableRegionWithReplica from 'testRollbackForSplitTableRegionWithReplica', {NAME => 'cf1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'testRollbackForSplitTableRegionWithReplica', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '2'}}, {NAME => 'cf1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-12 02:54:29,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] procedure2.ProcedureExecutor(1029): Stored pid=88, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-04-12 02:54:29,099 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackForSplitTableRegionWithReplica/.tabledesc/.tableinfo.0000000002 2023-04-12 02:54:29,101 DEBUG [PEWorker-1] util.FSTableDescriptors(519): Deleted hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackForSplitTableRegionWithReplica/.tabledesc/.tableinfo.0000000001 2023-04-12 02:54:29,102 INFO [PEWorker-1] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackForSplitTableRegionWithReplica/.tabledesc/.tableinfo.0000000002 2023-04-12 02:54:29,106 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=89, ppid=88, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=testRollbackForSplitTableRegionWithReplica}] 2023-04-12 02:54:29,126 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=90, ppid=89, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRollbackForSplitTableRegionWithReplica, region=4381c9c8e57c5dbb2533908abaf07cf1, REOPEN/MOVE}] 2023-04-12 02:54:29,131 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=90, ppid=89, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRollbackForSplitTableRegionWithReplica, region=4381c9c8e57c5dbb2533908abaf07cf1, REOPEN/MOVE 2023-04-12 02:54:29,132 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=90 updating hbase:meta row=4381c9c8e57c5dbb2533908abaf07cf1, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:29,132 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRollbackForSplitTableRegionWithReplica,,1681268068555.4381c9c8e57c5dbb2533908abaf07cf1.","families":{"info":[{"qualifier":"regioninfo","vlen":76,"tag":[],"timestamp":"1681268069132"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681268069132"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681268069132"}]},"ts":"1681268069132"} 2023-04-12 02:54:29,134 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=91, ppid=90, state=RUNNABLE; CloseRegionProcedure 4381c9c8e57c5dbb2533908abaf07cf1, server=jenkins-hbase4.apache.org,43091,1681268031804}] 2023-04-12 02:54:29,293 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 4381c9c8e57c5dbb2533908abaf07cf1 2023-04-12 02:54:29,294 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 4381c9c8e57c5dbb2533908abaf07cf1, disabling compactions & flushes 2023-04-12 02:54:29,294 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region testRollbackForSplitTableRegionWithReplica,,1681268068555.4381c9c8e57c5dbb2533908abaf07cf1. 2023-04-12 02:54:29,294 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testRollbackForSplitTableRegionWithReplica,,1681268068555.4381c9c8e57c5dbb2533908abaf07cf1. 2023-04-12 02:54:29,294 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on testRollbackForSplitTableRegionWithReplica,,1681268068555.4381c9c8e57c5dbb2533908abaf07cf1. after waiting 0 ms 2023-04-12 02:54:29,295 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region testRollbackForSplitTableRegionWithReplica,,1681268068555.4381c9c8e57c5dbb2533908abaf07cf1. 2023-04-12 02:54:29,298 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackForSplitTableRegionWithReplica/4381c9c8e57c5dbb2533908abaf07cf1/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-04-12 02:54:29,299 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro 2023-04-12 02:54:29,299 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed testRollbackForSplitTableRegionWithReplica,,1681268068555.4381c9c8e57c5dbb2533908abaf07cf1. 2023-04-12 02:54:29,299 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 4381c9c8e57c5dbb2533908abaf07cf1: 2023-04-12 02:54:29,299 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(3509): Not adding moved region record: 4381c9c8e57c5dbb2533908abaf07cf1 to self. 2023-04-12 02:54:29,300 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 4381c9c8e57c5dbb2533908abaf07cf1 2023-04-12 02:54:29,301 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=90 updating hbase:meta row=4381c9c8e57c5dbb2533908abaf07cf1, regionState=CLOSED 2023-04-12 02:54:29,301 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRollbackForSplitTableRegionWithReplica,,1681268068555.4381c9c8e57c5dbb2533908abaf07cf1.","families":{"info":[{"qualifier":"regioninfo","vlen":76,"tag":[],"timestamp":"1681268069301"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681268069301"}]},"ts":"1681268069301"} 2023-04-12 02:54:29,309 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=91, resume processing ppid=90 2023-04-12 02:54:29,309 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=91, ppid=90, state=SUCCESS; CloseRegionProcedure 4381c9c8e57c5dbb2533908abaf07cf1, server=jenkins-hbase4.apache.org,43091,1681268031804 in 170 msec 2023-04-12 02:54:29,310 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=90, ppid=89, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRollbackForSplitTableRegionWithReplica, region=4381c9c8e57c5dbb2533908abaf07cf1, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase4.apache.org,43091,1681268031804; forceNewPlan=false, retain=true 2023-04-12 02:54:29,460 INFO [jenkins-hbase4:40281] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-04-12 02:54:29,461 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=90 updating hbase:meta row=4381c9c8e57c5dbb2533908abaf07cf1, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:29,461 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRollbackForSplitTableRegionWithReplica,,1681268068555.4381c9c8e57c5dbb2533908abaf07cf1.","families":{"info":[{"qualifier":"regioninfo","vlen":76,"tag":[],"timestamp":"1681268069461"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681268069461"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681268069461"}]},"ts":"1681268069461"} 2023-04-12 02:54:29,463 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=92, ppid=90, state=RUNNABLE; OpenRegionProcedure 4381c9c8e57c5dbb2533908abaf07cf1, server=jenkins-hbase4.apache.org,43091,1681268031804}] 2023-04-12 02:54:29,619 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRollbackForSplitTableRegionWithReplica,,1681268068555.4381c9c8e57c5dbb2533908abaf07cf1. 2023-04-12 02:54:29,619 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 4381c9c8e57c5dbb2533908abaf07cf1, NAME => 'testRollbackForSplitTableRegionWithReplica,,1681268068555.4381c9c8e57c5dbb2533908abaf07cf1.', STARTKEY => '', ENDKEY => ''} 2023-04-12 02:54:29,619 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(174): System coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro loaded, priority=536870911. 2023-04-12 02:54:29,620 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRollbackForSplitTableRegionWithReplica 4381c9c8e57c5dbb2533908abaf07cf1 2023-04-12 02:54:29,620 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated testRollbackForSplitTableRegionWithReplica,,1681268068555.4381c9c8e57c5dbb2533908abaf07cf1.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-12 02:54:29,620 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 4381c9c8e57c5dbb2533908abaf07cf1 2023-04-12 02:54:29,620 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 4381c9c8e57c5dbb2533908abaf07cf1 2023-04-12 02:54:29,628 INFO [StoreOpener-4381c9c8e57c5dbb2533908abaf07cf1-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf1 of region 4381c9c8e57c5dbb2533908abaf07cf1 2023-04-12 02:54:29,630 DEBUG [StoreOpener-4381c9c8e57c5dbb2533908abaf07cf1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackForSplitTableRegionWithReplica/4381c9c8e57c5dbb2533908abaf07cf1/cf1 2023-04-12 02:54:29,630 DEBUG [StoreOpener-4381c9c8e57c5dbb2533908abaf07cf1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackForSplitTableRegionWithReplica/4381c9c8e57c5dbb2533908abaf07cf1/cf1 2023-04-12 02:54:29,631 INFO [StoreOpener-4381c9c8e57c5dbb2533908abaf07cf1-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 4381c9c8e57c5dbb2533908abaf07cf1 columnFamilyName cf1 2023-04-12 02:54:29,631 INFO [StoreOpener-4381c9c8e57c5dbb2533908abaf07cf1-1] regionserver.HStore(310): Store=4381c9c8e57c5dbb2533908abaf07cf1/cf1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-12 02:54:29,632 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackForSplitTableRegionWithReplica/4381c9c8e57c5dbb2533908abaf07cf1 2023-04-12 02:54:29,633 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackForSplitTableRegionWithReplica/4381c9c8e57c5dbb2533908abaf07cf1 2023-04-12 02:54:29,636 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 4381c9c8e57c5dbb2533908abaf07cf1 2023-04-12 02:54:29,637 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 4381c9c8e57c5dbb2533908abaf07cf1; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=65455260, jitterRate=-0.02464061975479126}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-04-12 02:54:29,637 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 4381c9c8e57c5dbb2533908abaf07cf1: 2023-04-12 02:54:29,638 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRollbackForSplitTableRegionWithReplica,,1681268068555.4381c9c8e57c5dbb2533908abaf07cf1., pid=92, masterSystemTime=1681268069616 2023-04-12 02:54:29,640 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRollbackForSplitTableRegionWithReplica,,1681268068555.4381c9c8e57c5dbb2533908abaf07cf1. 2023-04-12 02:54:29,640 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRollbackForSplitTableRegionWithReplica,,1681268068555.4381c9c8e57c5dbb2533908abaf07cf1. 2023-04-12 02:54:29,641 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=90 updating hbase:meta row=4381c9c8e57c5dbb2533908abaf07cf1, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:54:29,641 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRollbackForSplitTableRegionWithReplica,,1681268068555.4381c9c8e57c5dbb2533908abaf07cf1.","families":{"info":[{"qualifier":"regioninfo","vlen":76,"tag":[],"timestamp":"1681268069641"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1681268069641"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681268069641"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681268069641"}]},"ts":"1681268069641"} 2023-04-12 02:54:29,645 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=92, resume processing ppid=90 2023-04-12 02:54:29,645 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=92, ppid=90, state=SUCCESS; OpenRegionProcedure 4381c9c8e57c5dbb2533908abaf07cf1, server=jenkins-hbase4.apache.org,43091,1681268031804 in 180 msec 2023-04-12 02:54:29,646 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=90, resume processing ppid=89 2023-04-12 02:54:29,646 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=90, ppid=89, state=SUCCESS; TransitRegionStateProcedure table=testRollbackForSplitTableRegionWithReplica, region=4381c9c8e57c5dbb2533908abaf07cf1, REOPEN/MOVE in 519 msec 2023-04-12 02:54:29,655 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=89, resume processing ppid=88 2023-04-12 02:54:29,655 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=89, ppid=88, state=SUCCESS; ReopenTableRegionsProcedure table=testRollbackForSplitTableRegionWithReplica in 543 msec 2023-04-12 02:54:29,659 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=93, ppid=88, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRollbackForSplitTableRegionWithReplica, region=94855f8e18950d1c0c1e1d85df03f9ad, ASSIGN}] 2023-04-12 02:54:29,661 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=93, ppid=88, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRollbackForSplitTableRegionWithReplica, region=94855f8e18950d1c0c1e1d85df03f9ad, ASSIGN 2023-04-12 02:54:29,662 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=93, ppid=88, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRollbackForSplitTableRegionWithReplica, region=94855f8e18950d1c0c1e1d85df03f9ad, ASSIGN; state=OFFLINE, location=null; forceNewPlan=false, retain=false 2023-04-12 02:54:29,670 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'testRollbackForSplitTableRegionWithReplica' 2023-04-12 02:54:29,813 DEBUG [jenkins-hbase4:40281] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-04-12 02:54:29,814 DEBUG [jenkins-hbase4:40281] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-04-12 02:54:29,814 DEBUG [jenkins-hbase4:40281] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-04-12 02:54:29,814 DEBUG [jenkins-hbase4:40281] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-04-12 02:54:29,814 DEBUG [jenkins-hbase4:40281] balancer.BaseLoadBalancer$Cluster(378): Number of tables=3, number of hosts=1, number of racks=1 2023-04-12 02:54:29,816 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=93 updating hbase:meta row=94855f8e18950d1c0c1e1d85df03f9ad, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:54:29,816 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRollbackForSplitTableRegionWithReplica,,1681268068555.4381c9c8e57c5dbb2533908abaf07cf1.","families":{"info":[{"qualifier":"regioninfo","vlen":76,"tag":[],"timestamp":"1681268069816"},{"qualifier":"sn_0001","vlen":45,"tag":[],"timestamp":"1681268069816"},{"qualifier":"state_0001","vlen":7,"tag":[],"timestamp":"1681268069816"}]},"ts":"1681268069816"} 2023-04-12 02:54:29,820 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=94, ppid=93, state=RUNNABLE; OpenRegionProcedure 94855f8e18950d1c0c1e1d85df03f9ad, server=jenkins-hbase4.apache.org,34579,1681268031719}] 2023-04-12 02:54:29,975 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRollbackForSplitTableRegionWithReplica,,1681268068555_0001.94855f8e18950d1c0c1e1d85df03f9ad. 2023-04-12 02:54:29,975 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 94855f8e18950d1c0c1e1d85df03f9ad, NAME => 'testRollbackForSplitTableRegionWithReplica,,1681268068555_0001.94855f8e18950d1c0c1e1d85df03f9ad.', STARTKEY => '', ENDKEY => '', REPLICA_ID => 1} 2023-04-12 02:54:29,976 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(174): System coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro loaded, priority=536870911. 2023-04-12 02:54:29,976 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRollbackForSplitTableRegionWithReplica 94855f8e18950d1c0c1e1d85df03f9ad 2023-04-12 02:54:29,976 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated testRollbackForSplitTableRegionWithReplica,,1681268068555_0001.94855f8e18950d1c0c1e1d85df03f9ad.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-12 02:54:29,976 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 94855f8e18950d1c0c1e1d85df03f9ad 2023-04-12 02:54:29,976 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 94855f8e18950d1c0c1e1d85df03f9ad 2023-04-12 02:54:29,976 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:30,076 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:30,176 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:30,277 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:30,377 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:30,477 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:30,577 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:30,678 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:30,778 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:30,878 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:30,978 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:31,079 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:31,179 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:31,237 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-12 02:54:31,279 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:31,379 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:31,479 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:31,580 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:31,680 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:31,780 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:31,880 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:31,981 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:32,081 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:32,181 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:32,281 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:32,381 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:32,482 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:32,582 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:32,682 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:32,782 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:32,882 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:32,983 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:33,083 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:33,183 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:33,283 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:33,384 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:33,484 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:33,584 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:33,684 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:33,784 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:33,885 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:33,985 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:34,085 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:34,185 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:34,286 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:34,386 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:34,486 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:34,586 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:34,686 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:34,787 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:34,887 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:34,987 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:35,087 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:35,188 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:35,288 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:35,388 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:35,488 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:35,588 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:35,689 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:35,789 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:35,889 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:35,989 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:36,089 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:36,190 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:36,290 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:36,390 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:36,490 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:36,591 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:36,691 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:36,791 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:36,891 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:36,991 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:37,092 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:37,192 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:37,292 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:37,392 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:37,493 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:37,593 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:37,693 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:37,793 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:37,893 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:37,994 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:38,094 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:38,194 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:38,294 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:38,394 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:38,495 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:38,595 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:38,695 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:38,795 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:38,896 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:38,996 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:39,096 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:39,196 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:39,296 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:39,397 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:39,497 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:39,597 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:39,697 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:39,798 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:39,898 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:39,998 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:40,098 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:40,198 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:40,299 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:40,399 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:40,499 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:40,599 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:40,700 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:40,800 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:40,900 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:41,000 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:41,100 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:41,201 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:41,301 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:41,401 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:41,501 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:41,602 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:41,702 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:41,802 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:41,902 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:42,002 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:42,103 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:42,203 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:42,303 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:42,403 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:42,504 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:42,604 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:42,704 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:42,804 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:42,905 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:43,005 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:43,105 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:43,205 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:43,305 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:43,406 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:43,506 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:43,606 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:43,706 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:43,807 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:43,907 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:44,007 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:44,107 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:44,207 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:44,308 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:44,408 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:44,508 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:44,608 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:44,708 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:44,809 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:44,909 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:45,009 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:45,109 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:45,209 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:45,310 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:45,410 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:45,510 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:45,610 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:45,710 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:45,811 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:45,911 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:46,011 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:46,111 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:46,211 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:46,312 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:46,412 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:46,512 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:46,612 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:46,712 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:46,812 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:46,913 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:47,013 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:47,113 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:47,213 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:47,313 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:47,414 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:47,514 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:47,614 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:47,714 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:47,814 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:47,915 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:48,015 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:48,115 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:48,215 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:48,315 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:48,416 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:48,516 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:48,616 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:48,716 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:48,816 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:48,916 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:49,017 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:49,117 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:49,217 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:49,317 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:49,417 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:49,518 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:49,618 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:49,718 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:49,818 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:49,918 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:50,019 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:50,119 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:50,219 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:50,319 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:50,419 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:50,520 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:50,620 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:50,720 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:50,820 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:50,920 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:51,021 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:51,121 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:51,221 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:51,321 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:51,421 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:51,522 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:51,622 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:51,722 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:51,822 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:51,922 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:52,023 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:52,123 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:52,223 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:52,323 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:52,424 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:52,524 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:52,624 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:52,724 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:52,824 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:52,925 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:53,025 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:53,125 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:53,200 DEBUG [regionserver/jenkins-hbase4:0.Chore.1] throttle.PressureAwareCompactionThroughputController(103): CompactionPressure is 0.0, tune throughput to 50.00 MB/second 2023-04-12 02:54:53,212 DEBUG [regionserver/jenkins-hbase4:0.Chore.1] throttle.PressureAwareCompactionThroughputController(103): CompactionPressure is 0.0, tune throughput to 50.00 MB/second 2023-04-12 02:54:53,225 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:53,263 DEBUG [regionserver/jenkins-hbase4:0.Chore.1] throttle.PressureAwareCompactionThroughputController(103): CompactionPressure is 0.0, tune throughput to 50.00 MB/second 2023-04-12 02:54:53,325 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:53,426 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:53,526 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:53,626 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:53,726 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:53,826 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:53,927 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:54,027 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:54,127 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:54,227 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:54,327 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:54,428 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:54,528 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:54,628 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:54,728 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:54,828 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:54,929 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:55,029 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:55,129 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:55,229 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:55,329 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:55,430 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:55,530 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:55,630 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:55,730 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:55,830 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:55,931 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:56,031 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:56,131 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:56,231 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:56,331 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:56,432 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:56,532 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:56,632 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:56,732 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:56,832 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:56,933 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:57,033 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:57,133 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:57,234 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:57,334 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:57,434 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:57,534 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:57,635 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:57,735 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:57,835 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:57,935 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:58,035 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:58,136 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:58,236 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:58,336 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:58,436 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:58,537 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:58,637 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:58,737 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:58,837 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:58,938 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:59,038 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:59,138 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:59,238 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:59,338 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:59,439 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:59,539 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:59,639 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:59,739 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:59,839 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:54:59,940 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:00,040 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:00,141 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:00,241 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:00,341 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:00,441 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:00,542 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:00,642 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:00,742 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:00,842 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:00,942 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:01,043 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:01,143 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:01,243 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:01,343 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:01,444 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:01,544 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:01,644 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:01,744 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:01,845 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:01,945 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:02,045 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:02,145 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:02,245 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:02,346 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:02,446 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:02,546 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:02,646 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:02,747 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:02,847 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:02,947 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:03,047 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:03,148 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:03,248 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:03,348 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:03,448 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:03,548 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:03,649 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:03,749 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:03,849 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:03,949 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:04,049 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:04,150 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:04,250 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:04,350 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:04,450 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:04,550 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:04,651 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:04,751 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:04,851 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:04,951 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:05,052 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:05,152 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:05,252 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:05,352 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:05,452 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:05,553 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:05,653 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:05,753 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:05,853 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:05,954 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:06,054 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:06,154 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:06,254 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:06,355 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:06,455 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:06,555 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:06,655 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:06,755 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:06,856 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:06,956 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:07,056 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:07,156 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:07,257 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:07,357 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:07,457 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:07,557 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:07,657 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:07,758 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:07,858 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:07,958 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:08,058 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:08,158 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:08,259 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:08,359 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:08,459 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:08,559 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:08,660 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:08,760 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:08,860 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:08,960 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:09,060 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:09,161 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:09,261 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:09,361 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:09,461 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:09,562 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:09,662 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:09,762 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:09,862 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:09,962 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:10,063 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:10,163 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:10,263 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:10,363 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:10,464 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:10,564 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:10,664 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:10,764 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:10,865 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:10,965 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:11,065 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:11,165 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:11,266 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:11,366 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:11,466 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:11,566 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:11,666 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:11,767 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:11,867 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:11,967 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:12,067 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:12,168 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:12,268 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:12,368 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:12,468 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:12,568 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:12,669 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:12,769 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:12,869 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:12,969 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:13,069 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:13,170 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:13,270 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:13,370 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:13,470 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:13,571 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:13,671 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:13,771 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:13,871 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:13,971 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:14,072 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:14,172 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:14,272 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:14,372 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:14,473 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:14,573 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:14,673 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:14,773 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:14,874 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:14,974 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:15,074 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:15,174 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:15,274 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:15,375 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:15,475 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:15,575 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:15,675 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:15,776 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:15,876 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:15,976 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:16,076 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:16,176 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:16,277 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:16,377 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:16,477 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:16,577 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:16,678 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:16,778 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:16,878 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:16,978 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:17,078 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:17,179 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:17,279 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:17,379 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:17,479 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:17,579 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:17,680 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:17,780 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:17,880 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:17,980 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:18,080 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:18,181 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:18,281 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:18,381 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:18,481 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:18,581 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:18,682 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:18,782 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:18,882 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:18,982 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:19,082 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:19,183 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:19,283 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:19,383 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:19,483 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:19,583 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:19,684 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:19,784 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:19,884 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:19,984 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:20,084 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:20,185 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:20,285 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:20,385 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:20,485 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:20,585 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:20,686 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:20,786 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:20,886 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:20,986 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:21,086 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:21,187 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:21,287 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:21,387 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:21,487 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:21,587 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:21,687 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:21,788 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:21,888 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:21,988 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:22,088 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:22,188 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:22,289 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:22,389 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:22,489 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:22,589 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:22,689 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:22,790 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:22,890 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:22,990 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:23,090 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:23,190 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:23,291 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:23,391 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:23,491 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:23,591 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:23,691 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:23,791 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:23,892 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:23,992 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:24,092 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:24,192 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:24,293 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:24,393 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:24,493 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:24,593 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:24,693 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:24,793 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:24,894 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:24,994 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:25,094 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:25,194 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:25,294 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:25,395 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:25,495 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:25,595 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:25,695 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:25,795 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:25,896 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:25,996 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:26,096 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:26,196 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:26,296 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:26,397 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:26,497 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:26,597 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:26,697 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:26,797 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:26,898 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:26,998 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:27,098 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:27,198 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:27,299 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:27,399 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:27,499 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:27,599 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:27,699 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:27,799 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:27,900 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:28,000 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:28,100 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:28,200 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:28,300 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:28,401 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:28,501 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:28,601 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:28,701 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:28,801 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:28,902 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:29,002 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:29,102 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:29,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40281] master.MasterRpcServices(1227): Checking to see if procedure is done pid=88 2023-04-12 02:55:29,202 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:29,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40281] master.MasterRpcServices(1227): Checking to see if procedure is done pid=88 2023-04-12 02:55:29,302 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:29,403 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:29,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40281] master.MasterRpcServices(1227): Checking to see if procedure is done pid=88 2023-04-12 02:55:29,503 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:29,603 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:29,703 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:29,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40281] master.MasterRpcServices(1227): Checking to see if procedure is done pid=88 2023-04-12 02:55:29,803 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:29,904 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:30,004 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:30,104 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:30,204 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:30,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40281] master.MasterRpcServices(1227): Checking to see if procedure is done pid=88 2023-04-12 02:55:30,304 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:30,405 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:30,505 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:30,605 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:30,705 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:30,806 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:30,906 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:31,006 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:31,106 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:31,207 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:31,284 DEBUG [Listener at localhost/46667] assignment.SplitTableRegionProcedure(219): Splittable=true state=OPEN, location=jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:31,285 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(1029): Stored pid=95, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testRollbackForSplitTableRegionWithReplica, parent=4381c9c8e57c5dbb2533908abaf07cf1, daughterA=2e35c2f6b8767f438f484c700a605299, daughterB=8407ddb6fc7704d121cd7f8a674aefb6 2023-04-12 02:55:31,286 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=95, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testRollbackForSplitTableRegionWithReplica, parent=4381c9c8e57c5dbb2533908abaf07cf1, daughterA=2e35c2f6b8767f438f484c700a605299, daughterB=8407ddb6fc7704d121cd7f8a674aefb6 2023-04-12 02:55:31,286 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=95, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testRollbackForSplitTableRegionWithReplica, parent=4381c9c8e57c5dbb2533908abaf07cf1, daughterA=2e35c2f6b8767f438f484c700a605299, daughterB=8407ddb6fc7704d121cd7f8a674aefb6 2023-04-12 02:55:31,287 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=95, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testRollbackForSplitTableRegionWithReplica, parent=4381c9c8e57c5dbb2533908abaf07cf1, daughterA=2e35c2f6b8767f438f484c700a605299, daughterB=8407ddb6fc7704d121cd7f8a674aefb6 2023-04-12 02:55:31,289 ERROR [PEWorker-1] assignment.SplitTableRegionProcedure(347): Splitting 4381c9c8e57c5dbb2533908abaf07cf1, pid=95, state=RUNNABLE:SPLIT_TABLE_REGION_CLOSE_PARENT_REGION, locked=true; SplitTableRegionProcedure table=testRollbackForSplitTableRegionWithReplica, parent=4381c9c8e57c5dbb2533908abaf07cf1, daughterA=2e35c2f6b8767f438f484c700a605299, daughterB=8407ddb6fc7704d121cd7f8a674aefb6 org.apache.hadoop.hbase.HBaseIOException: The parent region state=OPENING, location=jenkins-hbase4.apache.org,34579,1681268031719, table=testRollbackForSplitTableRegionWithReplica, region=94855f8e18950d1c0c1e1d85df03f9ad is currently in transition, give up at org.apache.hadoop.hbase.master.assignment.AssignmentManagerUtil.createUnassignProceduresForSplitOrMerge(AssignmentManagerUtil.java:120) at org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure.createUnassignProcedures(SplitTableRegionProcedure.java:881) at org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure.executeFromState(SplitTableRegionProcedure.java:298) at org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure.executeFromState(SplitTableRegionProcedure.java:94) at org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:188) at org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:922) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1646) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-04-12 02:55:31,291 INFO [PEWorker-1] procedure2.ProcedureExecutor(1528): Rolled back pid=95, state=ROLLEDBACK, exception=org.apache.hadoop.hbase.HBaseIOException via master-split-regions:org.apache.hadoop.hbase.HBaseIOException: The parent region state=OPENING, location=jenkins-hbase4.apache.org,34579,1681268031719, table=testRollbackForSplitTableRegionWithReplica, region=94855f8e18950d1c0c1e1d85df03f9ad is currently in transition, give up; SplitTableRegionProcedure table=testRollbackForSplitTableRegionWithReplica, parent=4381c9c8e57c5dbb2533908abaf07cf1, daughterA=2e35c2f6b8767f438f484c700a605299, daughterB=8407ddb6fc7704d121cd7f8a674aefb6 exec-time=7 msec 2023-04-12 02:55:31,307 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:31,407 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:31,507 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-12 02:55:31,608 INFO [StoreOpener-94855f8e18950d1c0c1e1d85df03f9ad-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf1 of region 94855f8e18950d1c0c1e1d85df03f9ad 2023-04-12 02:55:31,612 DEBUG [StoreOpener-94855f8e18950d1c0c1e1d85df03f9ad-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackForSplitTableRegionWithReplica/4381c9c8e57c5dbb2533908abaf07cf1/cf1 2023-04-12 02:55:31,612 DEBUG [StoreOpener-94855f8e18950d1c0c1e1d85df03f9ad-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackForSplitTableRegionWithReplica/4381c9c8e57c5dbb2533908abaf07cf1/cf1 2023-04-12 02:55:31,612 INFO [StoreOpener-94855f8e18950d1c0c1e1d85df03f9ad-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 94855f8e18950d1c0c1e1d85df03f9ad columnFamilyName cf1 2023-04-12 02:55:31,613 INFO [StoreOpener-94855f8e18950d1c0c1e1d85df03f9ad-1] regionserver.HStore(310): Store=94855f8e18950d1c0c1e1d85df03f9ad/cf1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-12 02:55:31,615 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 94855f8e18950d1c0c1e1d85df03f9ad; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=68259193, jitterRate=0.01714123785495758}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-04-12 02:55:31,616 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 94855f8e18950d1c0c1e1d85df03f9ad: 2023-04-12 02:55:31,616 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRollbackForSplitTableRegionWithReplica,,1681268068555_0001.94855f8e18950d1c0c1e1d85df03f9ad., pid=94, masterSystemTime=1681268069972 2023-04-12 02:55:31,617 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRollbackForSplitTableRegionWithReplica,,1681268068555_0001.94855f8e18950d1c0c1e1d85df03f9ad. 2023-04-12 02:55:31,618 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRollbackForSplitTableRegionWithReplica,,1681268068555_0001.94855f8e18950d1c0c1e1d85df03f9ad. 2023-04-12 02:55:31,618 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=93 updating hbase:meta row=94855f8e18950d1c0c1e1d85df03f9ad, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:31,618 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRollbackForSplitTableRegionWithReplica,,1681268068555.4381c9c8e57c5dbb2533908abaf07cf1.","families":{"info":[{"qualifier":"regioninfo","vlen":76,"tag":[],"timestamp":"1681268131618"},{"qualifier":"server_0001","vlen":31,"tag":[],"timestamp":"1681268131618"},{"qualifier":"serverstartcode_0001","vlen":8,"tag":[],"timestamp":"1681268131618"},{"qualifier":"seqnumDuringOpen_0001","vlen":8,"tag":[],"timestamp":"1681268131618"}]},"ts":"1681268131618"} 2023-04-12 02:55:31,621 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=94, resume processing ppid=93 2023-04-12 02:55:31,622 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=94, ppid=93, state=SUCCESS; OpenRegionProcedure 94855f8e18950d1c0c1e1d85df03f9ad, server=jenkins-hbase4.apache.org,34579,1681268031719 in 1 mins, 1.802 sec 2023-04-12 02:55:31,623 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=93, resume processing ppid=88 2023-04-12 02:55:31,623 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=93, ppid=88, state=SUCCESS; TransitRegionStateProcedure table=testRollbackForSplitTableRegionWithReplica, region=94855f8e18950d1c0c1e1d85df03f9ad, ASSIGN in 1 mins, 1.963 sec 2023-04-12 02:55:31,626 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=88, state=SUCCESS; ModifyTableProcedure table=testRollbackForSplitTableRegionWithReplica in 1 mins, 2.555 sec 2023-04-12 02:55:31,627 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] ipc.RpcServer(528): (responseTooSlow): {"call":"ModifyTable(org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$ModifyTableRequest)","starttimems":"1681268069062","responsesize":"2","method":"ModifyTable","param":"TODO: class org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$ModifyTableRequest","processingtimems":62565,"client":"172.31.14.131:44732","queuetimems":0,"class":"HMaster"} 2023-04-12 02:55:31,628 DEBUG [RPCClient-NioEventLoopGroup-6-2] ipc.NettyRpcDuplexHandler(156): Unknown callId: 617, skipping over this response of 3 bytes 2023-04-12 02:55:33,536 INFO [Listener at localhost/46667] procedure2.ProcedureTestingUtility(335): procId=95 exception: org.apache.hadoop.hbase.HBaseIOException: The parent region state=OPENING, location=jenkins-hbase4.apache.org,34579,1681268031719, table=testRollbackForSplitTableRegionWithReplica, region=94855f8e18950d1c0c1e1d85df03f9ad is currently in transition, give up 2023-04-12 02:55:33,538 WARN [Listener at localhost/46667] procedure2.ProcedureTestingUtility(193): Set Kill before store update to: false 2023-04-12 02:55:33,540 INFO [Listener at localhost/46667] client.HBaseAdmin$15(890): Started disable of testRollbackForSplitTableRegionWithReplica 2023-04-12 02:55:33,540 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable testRollbackForSplitTableRegionWithReplica 2023-04-12 02:55:33,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] procedure2.ProcedureExecutor(1029): Stored pid=96, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-04-12 02:55:33,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] master.MasterRpcServices(1227): Checking to see if procedure is done pid=96 2023-04-12 02:55:33,545 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRollbackForSplitTableRegionWithReplica","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681268133545"}]},"ts":"1681268133545"} 2023-04-12 02:55:33,548 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testRollbackForSplitTableRegionWithReplica, state=DISABLING in hbase:meta 2023-04-12 02:55:33,550 INFO [PEWorker-1] procedure.DisableTableProcedure(293): Set testRollbackForSplitTableRegionWithReplica to state=DISABLING 2023-04-12 02:55:33,551 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=97, ppid=96, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRollbackForSplitTableRegionWithReplica, region=4381c9c8e57c5dbb2533908abaf07cf1, UNASSIGN}, {pid=98, ppid=96, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRollbackForSplitTableRegionWithReplica, region=94855f8e18950d1c0c1e1d85df03f9ad, UNASSIGN}] 2023-04-12 02:55:33,552 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=98, ppid=96, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRollbackForSplitTableRegionWithReplica, region=94855f8e18950d1c0c1e1d85df03f9ad, UNASSIGN 2023-04-12 02:55:33,553 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=98 updating hbase:meta row=94855f8e18950d1c0c1e1d85df03f9ad, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:33,553 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRollbackForSplitTableRegionWithReplica,,1681268068555.4381c9c8e57c5dbb2533908abaf07cf1.","families":{"info":[{"qualifier":"regioninfo","vlen":76,"tag":[],"timestamp":"1681268133553"},{"qualifier":"sn_0001","vlen":45,"tag":[],"timestamp":"1681268133553"},{"qualifier":"state_0001","vlen":7,"tag":[],"timestamp":"1681268133553"}]},"ts":"1681268133553"} 2023-04-12 02:55:33,554 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=99, ppid=98, state=RUNNABLE; CloseRegionProcedure 94855f8e18950d1c0c1e1d85df03f9ad, server=jenkins-hbase4.apache.org,34579,1681268031719}] 2023-04-12 02:55:33,556 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=97, ppid=96, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRollbackForSplitTableRegionWithReplica, region=4381c9c8e57c5dbb2533908abaf07cf1, UNASSIGN 2023-04-12 02:55:33,557 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=97 updating hbase:meta row=4381c9c8e57c5dbb2533908abaf07cf1, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:33,557 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRollbackForSplitTableRegionWithReplica,,1681268068555.4381c9c8e57c5dbb2533908abaf07cf1.","families":{"info":[{"qualifier":"regioninfo","vlen":76,"tag":[],"timestamp":"1681268133557"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681268133557"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681268133557"}]},"ts":"1681268133557"} 2023-04-12 02:55:33,558 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=100, ppid=97, state=RUNNABLE; CloseRegionProcedure 4381c9c8e57c5dbb2533908abaf07cf1, server=jenkins-hbase4.apache.org,43091,1681268031804}] 2023-04-12 02:55:33,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] master.MasterRpcServices(1227): Checking to see if procedure is done pid=96 2023-04-12 02:55:33,707 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 94855f8e18950d1c0c1e1d85df03f9ad 2023-04-12 02:55:33,708 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 94855f8e18950d1c0c1e1d85df03f9ad, disabling compactions & flushes 2023-04-12 02:55:33,708 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region testRollbackForSplitTableRegionWithReplica,,1681268068555_0001.94855f8e18950d1c0c1e1d85df03f9ad. 2023-04-12 02:55:33,708 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testRollbackForSplitTableRegionWithReplica,,1681268068555_0001.94855f8e18950d1c0c1e1d85df03f9ad. 2023-04-12 02:55:33,708 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on testRollbackForSplitTableRegionWithReplica,,1681268068555_0001.94855f8e18950d1c0c1e1d85df03f9ad. after waiting 0 ms 2023-04-12 02:55:33,708 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region testRollbackForSplitTableRegionWithReplica,,1681268068555_0001.94855f8e18950d1c0c1e1d85df03f9ad. 2023-04-12 02:55:33,708 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro 2023-04-12 02:55:33,709 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed testRollbackForSplitTableRegionWithReplica,,1681268068555_0001.94855f8e18950d1c0c1e1d85df03f9ad. 2023-04-12 02:55:33,709 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 94855f8e18950d1c0c1e1d85df03f9ad: 2023-04-12 02:55:33,711 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 94855f8e18950d1c0c1e1d85df03f9ad 2023-04-12 02:55:33,711 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=98 updating hbase:meta row=94855f8e18950d1c0c1e1d85df03f9ad, regionState=CLOSED 2023-04-12 02:55:33,711 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRollbackForSplitTableRegionWithReplica,,1681268068555.4381c9c8e57c5dbb2533908abaf07cf1.","families":{"info":[{"qualifier":"regioninfo","vlen":76,"tag":[],"timestamp":"1681268133711"},{"qualifier":"state_0001","vlen":6,"tag":[],"timestamp":"1681268133711"}]},"ts":"1681268133711"} 2023-04-12 02:55:33,712 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 4381c9c8e57c5dbb2533908abaf07cf1 2023-04-12 02:55:33,713 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 4381c9c8e57c5dbb2533908abaf07cf1, disabling compactions & flushes 2023-04-12 02:55:33,713 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region testRollbackForSplitTableRegionWithReplica,,1681268068555.4381c9c8e57c5dbb2533908abaf07cf1. 2023-04-12 02:55:33,713 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testRollbackForSplitTableRegionWithReplica,,1681268068555.4381c9c8e57c5dbb2533908abaf07cf1. 2023-04-12 02:55:33,713 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on testRollbackForSplitTableRegionWithReplica,,1681268068555.4381c9c8e57c5dbb2533908abaf07cf1. after waiting 0 ms 2023-04-12 02:55:33,713 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region testRollbackForSplitTableRegionWithReplica,,1681268068555.4381c9c8e57c5dbb2533908abaf07cf1. 2023-04-12 02:55:33,721 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRollbackForSplitTableRegionWithReplica/4381c9c8e57c5dbb2533908abaf07cf1/recovered.edits/7.seqid, newMaxSeqId=7, maxSeqId=4 2023-04-12 02:55:33,721 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=99, resume processing ppid=98 2023-04-12 02:55:33,722 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=99, ppid=98, state=SUCCESS; CloseRegionProcedure 94855f8e18950d1c0c1e1d85df03f9ad, server=jenkins-hbase4.apache.org,34579,1681268031719 in 160 msec 2023-04-12 02:55:33,723 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro 2023-04-12 02:55:33,723 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed testRollbackForSplitTableRegionWithReplica,,1681268068555.4381c9c8e57c5dbb2533908abaf07cf1. 2023-04-12 02:55:33,723 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 4381c9c8e57c5dbb2533908abaf07cf1: 2023-04-12 02:55:33,724 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=98, ppid=96, state=SUCCESS; TransitRegionStateProcedure table=testRollbackForSplitTableRegionWithReplica, region=94855f8e18950d1c0c1e1d85df03f9ad, UNASSIGN in 171 msec 2023-04-12 02:55:33,725 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 4381c9c8e57c5dbb2533908abaf07cf1 2023-04-12 02:55:33,725 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=97 updating hbase:meta row=4381c9c8e57c5dbb2533908abaf07cf1, regionState=CLOSED 2023-04-12 02:55:33,725 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRollbackForSplitTableRegionWithReplica,,1681268068555.4381c9c8e57c5dbb2533908abaf07cf1.","families":{"info":[{"qualifier":"regioninfo","vlen":76,"tag":[],"timestamp":"1681268133725"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681268133725"}]},"ts":"1681268133725"} 2023-04-12 02:55:33,731 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=100, resume processing ppid=97 2023-04-12 02:55:33,732 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=100, ppid=97, state=SUCCESS; CloseRegionProcedure 4381c9c8e57c5dbb2533908abaf07cf1, server=jenkins-hbase4.apache.org,43091,1681268031804 in 170 msec 2023-04-12 02:55:33,733 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=97, resume processing ppid=96 2023-04-12 02:55:33,733 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=97, ppid=96, state=SUCCESS; TransitRegionStateProcedure table=testRollbackForSplitTableRegionWithReplica, region=4381c9c8e57c5dbb2533908abaf07cf1, UNASSIGN in 181 msec 2023-04-12 02:55:33,735 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRollbackForSplitTableRegionWithReplica","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681268133735"}]},"ts":"1681268133735"} 2023-04-12 02:55:33,736 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testRollbackForSplitTableRegionWithReplica, state=DISABLED in hbase:meta 2023-04-12 02:55:33,738 INFO [PEWorker-1] procedure.DisableTableProcedure(305): Set testRollbackForSplitTableRegionWithReplica to state=DISABLED 2023-04-12 02:55:33,741 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=96, state=SUCCESS; DisableTableProcedure table=testRollbackForSplitTableRegionWithReplica in 198 msec 2023-04-12 02:55:33,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] master.MasterRpcServices(1227): Checking to see if procedure is done pid=96 2023-04-12 02:55:33,847 INFO [Listener at localhost/46667] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testRollbackForSplitTableRegionWithReplica, procId: 96 completed 2023-04-12 02:55:33,848 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete testRollbackForSplitTableRegionWithReplica 2023-04-12 02:55:33,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] procedure2.ProcedureExecutor(1029): Stored pid=101, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-04-12 02:55:33,858 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=101, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-04-12 02:55:33,859 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=101, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-04-12 02:55:33,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] master.MasterRpcServices(1227): Checking to see if procedure is done pid=101 2023-04-12 02:55:33,867 DEBUG [HFileArchiver-9] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testRollbackForSplitTableRegionWithReplica/4381c9c8e57c5dbb2533908abaf07cf1 2023-04-12 02:55:33,869 DEBUG [HFileArchiver-9] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testRollbackForSplitTableRegionWithReplica/4381c9c8e57c5dbb2533908abaf07cf1/cf1, FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testRollbackForSplitTableRegionWithReplica/4381c9c8e57c5dbb2533908abaf07cf1/recovered.edits] 2023-04-12 02:55:33,875 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testRollbackForSplitTableRegionWithReplica/4381c9c8e57c5dbb2533908abaf07cf1/recovered.edits/7.seqid to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testRollbackForSplitTableRegionWithReplica/4381c9c8e57c5dbb2533908abaf07cf1/recovered.edits/7.seqid 2023-04-12 02:55:33,876 DEBUG [HFileArchiver-9] backup.HFileArchiver(596): Deleted hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testRollbackForSplitTableRegionWithReplica/4381c9c8e57c5dbb2533908abaf07cf1 2023-04-12 02:55:33,877 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived testRollbackForSplitTableRegionWithReplica regions 2023-04-12 02:55:33,880 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=101, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-04-12 02:55:33,882 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of testRollbackForSplitTableRegionWithReplica from hbase:meta 2023-04-12 02:55:33,884 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'testRollbackForSplitTableRegionWithReplica' descriptor. 2023-04-12 02:55:33,885 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=101, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-04-12 02:55:33,885 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'testRollbackForSplitTableRegionWithReplica' from region states. 2023-04-12 02:55:33,885 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRollbackForSplitTableRegionWithReplica,,1681268068555.4381c9c8e57c5dbb2533908abaf07cf1.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1681268133885"}]},"ts":"9223372036854775807"} 2023-04-12 02:55:33,885 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRollbackForSplitTableRegionWithReplica,,1681268068555_0001.94855f8e18950d1c0c1e1d85df03f9ad.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1681268133885"}]},"ts":"9223372036854775807"} 2023-04-12 02:55:33,891 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 2 regions from META 2023-04-12 02:55:33,891 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 4381c9c8e57c5dbb2533908abaf07cf1, NAME => 'testRollbackForSplitTableRegionWithReplica,,1681268068555.4381c9c8e57c5dbb2533908abaf07cf1.', STARTKEY => '', ENDKEY => ''}, {ENCODED => 94855f8e18950d1c0c1e1d85df03f9ad, NAME => 'testRollbackForSplitTableRegionWithReplica,,1681268068555_0001.94855f8e18950d1c0c1e1d85df03f9ad.', STARTKEY => '', ENDKEY => '', REPLICA_ID => 1}] 2023-04-12 02:55:33,891 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'testRollbackForSplitTableRegionWithReplica' as deleted. 2023-04-12 02:55:33,891 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRollbackForSplitTableRegionWithReplica","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1681268133891"}]},"ts":"9223372036854775807"} 2023-04-12 02:55:33,893 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table testRollbackForSplitTableRegionWithReplica state from META 2023-04-12 02:55:33,896 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=101, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-04-12 02:55:33,897 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=101, state=SUCCESS; DeleteTableProcedure table=testRollbackForSplitTableRegionWithReplica in 48 msec 2023-04-12 02:55:33,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] master.MasterRpcServices(1227): Checking to see if procedure is done pid=101 2023-04-12 02:55:33,964 INFO [Listener at localhost/46667] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:testRollbackForSplitTableRegionWithReplica, procId: 101 completed 2023-04-12 02:55:33,980 INFO [Listener at localhost/46667] hbase.ResourceChecker(175): after: master.assignment.TestSplitTableRegionProcedure#testRollbackForSplitTableRegionWithReplica Thread=344 (was 408), OpenFileDescriptor=649 (was 835), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=314 (was 423), ProcessCount=172 (was 172), AvailableMemoryMB=844 (was 1038) 2023-04-12 02:55:33,995 INFO [Listener at localhost/46667] hbase.ResourceChecker(147): before: master.assignment.TestSplitTableRegionProcedure#testInvalidSplitKey Thread=345, OpenFileDescriptor=649, MaxFileDescriptor=60000, SystemLoadAverage=314, ProcessCount=172, AvailableMemoryMB=843 2023-04-12 02:55:33,995 WARN [Listener at localhost/46667] procedure2.ProcedureTestingUtility(193): Set Kill before store update to: false 2023-04-12 02:55:33,998 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/balancer 2023-04-12 02:55:33,998 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] master.MasterRpcServices(492): Client=jenkins//172.31.14.131 set balanceSwitch=false 2023-04-12 02:55:33,999 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(1029): Stored pid=102, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=testInvalidSplitKey 2023-04-12 02:55:34,001 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=102, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=testInvalidSplitKey execute state=CREATE_TABLE_PRE_OPERATION 2023-04-12 02:55:34,002 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=102, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=testInvalidSplitKey execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-12 02:55:34,003 DEBUG [HFileArchiver-10] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testInvalidSplitKey/5add8dc92f700932b95abfa5a81f714d 2023-04-12 02:55:34,004 DEBUG [HFileArchiver-10] backup.HFileArchiver(153): Directory hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testInvalidSplitKey/5add8dc92f700932b95abfa5a81f714d empty. 2023-04-12 02:55:34,004 DEBUG [HFileArchiver-10] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testInvalidSplitKey/5add8dc92f700932b95abfa5a81f714d 2023-04-12 02:55:34,004 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived testInvalidSplitKey regions 2023-04-12 02:55:34,020 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testInvalidSplitKey/.tabledesc/.tableinfo.0000000001 2023-04-12 02:55:34,027 INFO [RegionOpenAndInit-testInvalidSplitKey-pool-0] regionserver.HRegion(7675): creating {ENCODED => 5add8dc92f700932b95abfa5a81f714d, NAME => 'testInvalidSplitKey,,1681268133998.5add8dc92f700932b95abfa5a81f714d.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='testInvalidSplitKey', {NAME => 'cf1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'cf2', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp 2023-04-12 02:55:34,036 DEBUG [RegionOpenAndInit-testInvalidSplitKey-pool-0] regionserver.HRegion(866): Instantiated testInvalidSplitKey,,1681268133998.5add8dc92f700932b95abfa5a81f714d.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-12 02:55:34,036 DEBUG [RegionOpenAndInit-testInvalidSplitKey-pool-0] regionserver.HRegion(1604): Closing 5add8dc92f700932b95abfa5a81f714d, disabling compactions & flushes 2023-04-12 02:55:34,036 INFO [RegionOpenAndInit-testInvalidSplitKey-pool-0] regionserver.HRegion(1626): Closing region testInvalidSplitKey,,1681268133998.5add8dc92f700932b95abfa5a81f714d. 2023-04-12 02:55:34,036 DEBUG [RegionOpenAndInit-testInvalidSplitKey-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testInvalidSplitKey,,1681268133998.5add8dc92f700932b95abfa5a81f714d. 2023-04-12 02:55:34,036 DEBUG [RegionOpenAndInit-testInvalidSplitKey-pool-0] regionserver.HRegion(1714): Acquired close lock on testInvalidSplitKey,,1681268133998.5add8dc92f700932b95abfa5a81f714d. after waiting 0 ms 2023-04-12 02:55:34,036 DEBUG [RegionOpenAndInit-testInvalidSplitKey-pool-0] regionserver.HRegion(1724): Updates disabled for region testInvalidSplitKey,,1681268133998.5add8dc92f700932b95abfa5a81f714d. 2023-04-12 02:55:34,036 INFO [RegionOpenAndInit-testInvalidSplitKey-pool-0] regionserver.HRegion(1838): Closed testInvalidSplitKey,,1681268133998.5add8dc92f700932b95abfa5a81f714d. 2023-04-12 02:55:34,036 DEBUG [RegionOpenAndInit-testInvalidSplitKey-pool-0] regionserver.HRegion(1558): Region close journal for 5add8dc92f700932b95abfa5a81f714d: 2023-04-12 02:55:34,038 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=102, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=testInvalidSplitKey execute state=CREATE_TABLE_ADD_TO_META 2023-04-12 02:55:34,039 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testInvalidSplitKey,,1681268133998.5add8dc92f700932b95abfa5a81f714d.","families":{"info":[{"qualifier":"regioninfo","vlen":53,"tag":[],"timestamp":"1681268134039"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681268134039"}]},"ts":"1681268134039"} 2023-04-12 02:55:34,040 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-12 02:55:34,041 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=102, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=testInvalidSplitKey execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-12 02:55:34,041 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testInvalidSplitKey","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681268134041"}]},"ts":"1681268134041"} 2023-04-12 02:55:34,042 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testInvalidSplitKey, state=ENABLING in hbase:meta 2023-04-12 02:55:34,047 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-04-12 02:55:34,048 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-04-12 02:55:34,048 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-04-12 02:55:34,048 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-04-12 02:55:34,048 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-04-12 02:55:34,048 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=103, ppid=102, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testInvalidSplitKey, region=5add8dc92f700932b95abfa5a81f714d, ASSIGN}] 2023-04-12 02:55:34,050 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=103, ppid=102, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testInvalidSplitKey, region=5add8dc92f700932b95abfa5a81f714d, ASSIGN 2023-04-12 02:55:34,051 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=103, ppid=102, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testInvalidSplitKey, region=5add8dc92f700932b95abfa5a81f714d, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,34579,1681268031719; forceNewPlan=false, retain=false 2023-04-12 02:55:34,201 INFO [jenkins-hbase4:40281] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-04-12 02:55:34,203 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=103 updating hbase:meta row=5add8dc92f700932b95abfa5a81f714d, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:34,203 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testInvalidSplitKey,,1681268133998.5add8dc92f700932b95abfa5a81f714d.","families":{"info":[{"qualifier":"regioninfo","vlen":53,"tag":[],"timestamp":"1681268134203"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681268134203"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681268134203"}]},"ts":"1681268134203"} 2023-04-12 02:55:34,205 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=104, ppid=103, state=RUNNABLE; OpenRegionProcedure 5add8dc92f700932b95abfa5a81f714d, server=jenkins-hbase4.apache.org,34579,1681268031719}] 2023-04-12 02:55:34,361 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testInvalidSplitKey,,1681268133998.5add8dc92f700932b95abfa5a81f714d. 2023-04-12 02:55:34,361 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 5add8dc92f700932b95abfa5a81f714d, NAME => 'testInvalidSplitKey,,1681268133998.5add8dc92f700932b95abfa5a81f714d.', STARTKEY => '', ENDKEY => ''} 2023-04-12 02:55:34,361 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(174): System coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro loaded, priority=536870911. 2023-04-12 02:55:34,361 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testInvalidSplitKey 5add8dc92f700932b95abfa5a81f714d 2023-04-12 02:55:34,361 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated testInvalidSplitKey,,1681268133998.5add8dc92f700932b95abfa5a81f714d.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-12 02:55:34,361 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 5add8dc92f700932b95abfa5a81f714d 2023-04-12 02:55:34,361 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 5add8dc92f700932b95abfa5a81f714d 2023-04-12 02:55:34,363 INFO [StoreOpener-5add8dc92f700932b95abfa5a81f714d-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf1 of region 5add8dc92f700932b95abfa5a81f714d 2023-04-12 02:55:34,368 DEBUG [StoreOpener-5add8dc92f700932b95abfa5a81f714d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testInvalidSplitKey/5add8dc92f700932b95abfa5a81f714d/cf1 2023-04-12 02:55:34,368 DEBUG [StoreOpener-5add8dc92f700932b95abfa5a81f714d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testInvalidSplitKey/5add8dc92f700932b95abfa5a81f714d/cf1 2023-04-12 02:55:34,368 INFO [StoreOpener-5add8dc92f700932b95abfa5a81f714d-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 5add8dc92f700932b95abfa5a81f714d columnFamilyName cf1 2023-04-12 02:55:34,369 INFO [StoreOpener-5add8dc92f700932b95abfa5a81f714d-1] regionserver.HStore(310): Store=5add8dc92f700932b95abfa5a81f714d/cf1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-12 02:55:34,369 INFO [StoreOpener-5add8dc92f700932b95abfa5a81f714d-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf2 of region 5add8dc92f700932b95abfa5a81f714d 2023-04-12 02:55:34,377 DEBUG [StoreOpener-5add8dc92f700932b95abfa5a81f714d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testInvalidSplitKey/5add8dc92f700932b95abfa5a81f714d/cf2 2023-04-12 02:55:34,377 DEBUG [StoreOpener-5add8dc92f700932b95abfa5a81f714d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testInvalidSplitKey/5add8dc92f700932b95abfa5a81f714d/cf2 2023-04-12 02:55:34,377 INFO [StoreOpener-5add8dc92f700932b95abfa5a81f714d-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 5add8dc92f700932b95abfa5a81f714d columnFamilyName cf2 2023-04-12 02:55:34,378 INFO [StoreOpener-5add8dc92f700932b95abfa5a81f714d-1] regionserver.HStore(310): Store=5add8dc92f700932b95abfa5a81f714d/cf2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-12 02:55:34,380 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testInvalidSplitKey/5add8dc92f700932b95abfa5a81f714d 2023-04-12 02:55:34,381 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testInvalidSplitKey/5add8dc92f700932b95abfa5a81f714d 2023-04-12 02:55:34,382 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testInvalidSplitKey descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-04-12 02:55:34,383 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 5add8dc92f700932b95abfa5a81f714d 2023-04-12 02:55:34,385 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testInvalidSplitKey/5add8dc92f700932b95abfa5a81f714d/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-12 02:55:34,386 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 5add8dc92f700932b95abfa5a81f714d; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=64052779, jitterRate=-0.045539215207099915}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-04-12 02:55:34,386 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 5add8dc92f700932b95abfa5a81f714d: 2023-04-12 02:55:34,387 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testInvalidSplitKey,,1681268133998.5add8dc92f700932b95abfa5a81f714d., pid=104, masterSystemTime=1681268134357 2023-04-12 02:55:34,388 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testInvalidSplitKey,,1681268133998.5add8dc92f700932b95abfa5a81f714d. 2023-04-12 02:55:34,388 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testInvalidSplitKey,,1681268133998.5add8dc92f700932b95abfa5a81f714d. 2023-04-12 02:55:34,389 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=103 updating hbase:meta row=5add8dc92f700932b95abfa5a81f714d, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:34,389 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testInvalidSplitKey,,1681268133998.5add8dc92f700932b95abfa5a81f714d.","families":{"info":[{"qualifier":"regioninfo","vlen":53,"tag":[],"timestamp":"1681268134389"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1681268134389"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681268134389"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681268134389"}]},"ts":"1681268134389"} 2023-04-12 02:55:34,393 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=104, resume processing ppid=103 2023-04-12 02:55:34,393 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=104, ppid=103, state=SUCCESS; OpenRegionProcedure 5add8dc92f700932b95abfa5a81f714d, server=jenkins-hbase4.apache.org,34579,1681268031719 in 185 msec 2023-04-12 02:55:34,394 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=103, resume processing ppid=102 2023-04-12 02:55:34,394 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=103, ppid=102, state=SUCCESS; TransitRegionStateProcedure table=testInvalidSplitKey, region=5add8dc92f700932b95abfa5a81f714d, ASSIGN in 345 msec 2023-04-12 02:55:34,395 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=102, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=testInvalidSplitKey execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-12 02:55:34,395 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testInvalidSplitKey","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681268134395"}]},"ts":"1681268134395"} 2023-04-12 02:55:34,397 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testInvalidSplitKey, state=ENABLED in hbase:meta 2023-04-12 02:55:34,399 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=102, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=testInvalidSplitKey execute state=CREATE_TABLE_POST_OPERATION 2023-04-12 02:55:34,400 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=102, state=SUCCESS; CreateTableProcedure table=testInvalidSplitKey in 400 msec 2023-04-12 02:55:34,511 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 02:55:34,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] procedure.ProcedureCoordinator(165): Submitting procedure testInvalidSplitKey 2023-04-12 02:55:34,513 INFO [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testInvalidSplitKey' 2023-04-12 02:55:34,513 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:55:34,513 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testInvalidSplitKey' starting 'acquire' 2023-04-12 02:55:34,513 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testInvalidSplitKey', kicking off acquire phase on members. 2023-04-12 02:55:34,514 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-12 02:55:34,514 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-04-12 02:55:34,515 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:34,515 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:55:34,515 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:55:34,515 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:55:34,515 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:55:34,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:55:34,515 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:55:34,515 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:34,515 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 02:55:34,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:55:34,515 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:55:34,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:55:34,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-04-12 02:55:34,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-04-12 02:55:34,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-04-12 02:55:34,516 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-12 02:55:34,516 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-12 02:55:34,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 02:55:34,518 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-12 02:55:34,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-04-12 02:55:34,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 02:55:34,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-04-12 02:55:34,518 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testInvalidSplitKey 2023-04-12 02:55:34,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 02:55:34,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-04-12 02:55:34,518 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testInvalidSplitKey 2023-04-12 02:55:34,518 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testInvalidSplitKey 2023-04-12 02:55:34,527 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testInvalidSplitKey 2023-04-12 02:55:34,527 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testInvalidSplitKey' with timeout 60000ms 2023-04-12 02:55:34,527 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testInvalidSplitKey 2023-04-12 02:55:34,527 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:55:34,527 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testInvalidSplitKey' with timeout 60000ms 2023-04-12 02:55:34,527 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:55:34,527 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testInvalidSplitKey' starting 'acquire' stage 2023-04-12 02:55:34,527 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testInvalidSplitKey' starting 'acquire' stage 2023-04-12 02:55:34,527 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testInvalidSplitKey' locally acquired 2023-04-12 02:55:34,527 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43091,1681268031804' joining acquired barrier for procedure (testInvalidSplitKey) in zk 2023-04-12 02:55:34,529 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-12 02:55:34,530 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-12 02:55:34,530 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testInvalidSplitKey' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 02:55:34,532 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testInvalidSplitKey 2023-04-12 02:55:34,532 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 02:55:34,532 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 02:55:34,532 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testInvalidSplitKey' with timeout 60000ms 2023-04-12 02:55:34,532 DEBUG [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testInvalidSplitKey,,1681268133998.5add8dc92f700932b95abfa5a81f714d. 2023-04-12 02:55:34,532 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:55:34,532 DEBUG [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testInvalidSplitKey,,1681268133998.5add8dc92f700932b95abfa5a81f714d. started... 2023-04-12 02:55:34,532 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testInvalidSplitKey' starting 'acquire' stage 2023-04-12 02:55:34,532 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testInvalidSplitKey' locally acquired 2023-04-12 02:55:34,532 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44679,1681268031772' joining acquired barrier for procedure (testInvalidSplitKey) in zk 2023-04-12 02:55:34,533 INFO [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5add8dc92f700932b95abfa5a81f714d 2/2 column families, dataSize=120 B heapSize=928 B 2023-04-12 02:55:34,536 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-12 02:55:34,537 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-12 02:55:34,537 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testInvalidSplitKey' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 02:55:34,551 INFO [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=60 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testInvalidSplitKey/5add8dc92f700932b95abfa5a81f714d/.tmp/cf1/638abb3c0a454a38824724abf4d1490a 2023-04-12 02:55:34,575 INFO [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=60 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testInvalidSplitKey/5add8dc92f700932b95abfa5a81f714d/.tmp/cf2/7e77ef73cda643b5b2908483142515d6 2023-04-12 02:55:34,581 DEBUG [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testInvalidSplitKey/5add8dc92f700932b95abfa5a81f714d/.tmp/cf1/638abb3c0a454a38824724abf4d1490a as hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testInvalidSplitKey/5add8dc92f700932b95abfa5a81f714d/cf1/638abb3c0a454a38824724abf4d1490a 2023-04-12 02:55:34,586 INFO [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testInvalidSplitKey/5add8dc92f700932b95abfa5a81f714d/cf1/638abb3c0a454a38824724abf4d1490a, entries=2, sequenceid=6, filesize=4.8 K 2023-04-12 02:55:34,588 DEBUG [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testInvalidSplitKey/5add8dc92f700932b95abfa5a81f714d/.tmp/cf2/7e77ef73cda643b5b2908483142515d6 as hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testInvalidSplitKey/5add8dc92f700932b95abfa5a81f714d/cf2/7e77ef73cda643b5b2908483142515d6 2023-04-12 02:55:34,593 INFO [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testInvalidSplitKey/5add8dc92f700932b95abfa5a81f714d/cf2/7e77ef73cda643b5b2908483142515d6, entries=2, sequenceid=6, filesize=4.8 K 2023-04-12 02:55:34,594 INFO [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~120 B/120, heapSize ~896 B/896, currentSize=0 B/0 for 5add8dc92f700932b95abfa5a81f714d in 61ms, sequenceid=6, compaction requested=false 2023-04-12 02:55:34,594 DEBUG [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'testInvalidSplitKey' 2023-04-12 02:55:34,595 DEBUG [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5add8dc92f700932b95abfa5a81f714d: 2023-04-12 02:55:34,595 DEBUG [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testInvalidSplitKey,,1681268133998.5add8dc92f700932b95abfa5a81f714d. 2023-04-12 02:55:34,595 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 02:55:34,595 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 02:55:34,595 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:34,595 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testInvalidSplitKey' locally acquired 2023-04-12 02:55:34,595 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34579,1681268031719' joining acquired barrier for procedure (testInvalidSplitKey) in zk 2023-04-12 02:55:34,597 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:34,597 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-12 02:55:34,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:34,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 02:55:34,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 02:55:34,597 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-12 02:55:34,597 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testInvalidSplitKey' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 02:55:34,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 02:55:34,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 02:55:34,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testInvalidSplitKey 2023-04-12 02:55:34,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:34,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:34,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:34,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 02:55:34,601 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34579,1681268031719' joining acquired barrier for procedure 'testInvalidSplitKey' on coordinator 2023-04-12 02:55:34,601 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5eda938b[Count = 0] remaining members to acquire global barrier 2023-04-12 02:55:34,601 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testInvalidSplitKey' starting 'in-barrier' execution. 2023-04-12 02:55:34,601 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-12 02:55:34,602 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-12 02:55:34,602 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-12 02:55:34,602 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-12 02:55:34,602 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-12 02:55:34,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-12 02:55:34,602 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:34,603 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 02:55:34,602 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-12 02:55:34,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-12 02:55:34,603 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testInvalidSplitKey' received 'reached' from coordinator. 2023-04-12 02:55:34,602 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-12 02:55:34,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-12 02:55:34,603 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testInvalidSplitKey' locally completed 2023-04-12 02:55:34,603 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testInvalidSplitKey' received 'reached' from coordinator. 2023-04-12 02:55:34,603 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testInvalidSplitKey' locally completed 2023-04-12 02:55:34,603 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testInvalidSplitKey' completed for member 'jenkins-hbase4.apache.org,44679,1681268031772' in zk 2023-04-12 02:55:34,603 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testInvalidSplitKey' completed for member 'jenkins-hbase4.apache.org,43091,1681268031804' in zk 2023-04-12 02:55:34,603 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testInvalidSplitKey' received 'reached' from coordinator. 2023-04-12 02:55:34,603 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testInvalidSplitKey' locally completed 2023-04-12 02:55:34,603 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testInvalidSplitKey' completed for member 'jenkins-hbase4.apache.org,34579,1681268031719' in zk 2023-04-12 02:55:34,604 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testInvalidSplitKey' has notified controller of completion 2023-04-12 02:55:34,604 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:55:34,605 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testInvalidSplitKey' completed. 2023-04-12 02:55:34,606 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testInvalidSplitKey' has notified controller of completion 2023-04-12 02:55:34,606 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:55:34,606 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testInvalidSplitKey' has notified controller of completion 2023-04-12 02:55:34,606 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testInvalidSplitKey' completed. 2023-04-12 02:55:34,606 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:55:34,606 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testInvalidSplitKey' completed. 2023-04-12 02:55:34,606 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:34,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:34,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 02:55:34,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 02:55:34,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 02:55:34,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 02:55:34,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testInvalidSplitKey 2023-04-12 02:55:34,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:34,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:34,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:34,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 02:55:34,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testInvalidSplitKey 2023-04-12 02:55:34,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:34,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:34,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:34,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testInvalidSplitKey' member 'jenkins-hbase4.apache.org,34579,1681268031719': 2023-04-12 02:55:34,610 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34579,1681268031719' released barrier for procedure'testInvalidSplitKey', counting down latch. Waiting for 0 more 2023-04-12 02:55:34,610 INFO [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testInvalidSplitKey' execution completed 2023-04-12 02:55:34,610 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 02:55:34,610 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 02:55:34,610 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testInvalidSplitKey 2023-04-12 02:55:34,610 INFO [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testInvalidSplitKeyincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 02:55:34,612 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-12 02:55:34,612 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:55:34,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-12 02:55:34,612 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-12 02:55:34,612 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-12 02:55:34,612 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:55:34,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-12 02:55:34,612 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:55:34,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-12 02:55:34,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-12 02:55:34,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-12 02:55:34,612 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-12 02:55:34,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-12 02:55:34,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-12 02:55:34,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 02:55:34,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 02:55:34,613 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:55:34,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:55:34,613 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:55:34,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:55:34,613 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:55:34,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:55:34,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 02:55:34,613 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:34,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-12 02:55:34,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-12 02:55:34,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-12 02:55:34,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testInvalidSplitKey 2023-04-12 02:55:34,614 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:34,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 02:55:34,614 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:34,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testInvalidSplitKey 2023-04-12 02:55:34,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:34,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:34,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:34,616 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:34,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 02:55:34,616 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:34,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testInvalidSplitKey 2023-04-12 02:55:34,616 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:34,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:34,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:34,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:34,620 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:55:34,620 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:55:34,620 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:34,620 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-04-12 02:55:34,620 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:34,620 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:34,620 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-04-12 02:55:34,620 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:34,620 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-12 02:55:34,620 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:34,620 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:55:34,620 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:55:34,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:55:34,620 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:55:34,620 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testInvalidSplitKey' 2023-04-12 02:55:34,621 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 02:55:34,620 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:34,621 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-12 02:55:34,621 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-12 02:55:34,620 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:55:34,621 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:55:34,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:55:34,620 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:55:34,621 DEBUG [Listener at localhost/46667] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testInvalidSplitKey'' to complete. (max 20000 ms per retry) 2023-04-12 02:55:34,621 DEBUG [Listener at localhost/46667] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 02:55:34,620 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:55:34,621 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:55:34,621 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:55:34,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:55:34,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:55:34,622 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:55:34,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:55:34,622 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:55:34,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:55:34,721 DEBUG [Listener at localhost/46667] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 02:55:34,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 02:55:34,739 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 02:55:34,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] procedure.ProcedureCoordinator(143): Procedure testInvalidSplitKey was in running list but was completed. Accepting new attempt. 2023-04-12 02:55:34,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] procedure.ProcedureCoordinator(165): Submitting procedure testInvalidSplitKey 2023-04-12 02:55:34,741 INFO [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testInvalidSplitKey' 2023-04-12 02:55:34,741 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:55:34,741 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testInvalidSplitKey' starting 'acquire' 2023-04-12 02:55:34,741 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testInvalidSplitKey', kicking off acquire phase on members. 2023-04-12 02:55:34,742 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-12 02:55:34,742 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-04-12 02:55:34,743 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:55:34,743 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:55:34,743 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:34,744 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:55:34,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:55:34,743 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:55:34,744 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:55:34,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:55:34,744 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:55:34,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:55:34,744 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:34,744 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 02:55:34,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-04-12 02:55:34,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-04-12 02:55:34,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-04-12 02:55:34,744 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-12 02:55:34,745 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-12 02:55:34,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 02:55:34,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-04-12 02:55:34,745 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-12 02:55:34,745 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testInvalidSplitKey 2023-04-12 02:55:34,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 02:55:34,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-04-12 02:55:34,745 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testInvalidSplitKey 2023-04-12 02:55:34,745 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testInvalidSplitKey is still present, removing 2023-04-12 02:55:34,745 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testInvalidSplitKey 2023-04-12 02:55:34,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 02:55:34,746 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testInvalidSplitKey is still present, removing 2023-04-12 02:55:34,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-04-12 02:55:34,746 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testInvalidSplitKey' with timeout 60000ms 2023-04-12 02:55:34,746 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testInvalidSplitKey 2023-04-12 02:55:34,746 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testInvalidSplitKey 2023-04-12 02:55:34,746 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:55:34,746 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testInvalidSplitKey' with timeout 60000ms 2023-04-12 02:55:34,746 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testInvalidSplitKey' starting 'acquire' stage 2023-04-12 02:55:34,746 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testInvalidSplitKey is still present, removing 2023-04-12 02:55:34,746 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:55:34,746 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testInvalidSplitKey 2023-04-12 02:55:34,746 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testInvalidSplitKey' locally acquired 2023-04-12 02:55:34,746 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testInvalidSplitKey' with timeout 60000ms 2023-04-12 02:55:34,746 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:55:34,746 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testInvalidSplitKey' starting 'acquire' stage 2023-04-12 02:55:34,746 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testInvalidSplitKey' starting 'acquire' stage 2023-04-12 02:55:34,746 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testInvalidSplitKey' locally acquired 2023-04-12 02:55:34,746 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43091,1681268031804' joining acquired barrier for procedure (testInvalidSplitKey) in zk 2023-04-12 02:55:34,746 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 02:55:34,746 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44679,1681268031772' joining acquired barrier for procedure (testInvalidSplitKey) in zk 2023-04-12 02:55:34,746 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 02:55:34,748 DEBUG [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testInvalidSplitKey,,1681268133998.5add8dc92f700932b95abfa5a81f714d. 2023-04-12 02:55:34,748 DEBUG [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testInvalidSplitKey,,1681268133998.5add8dc92f700932b95abfa5a81f714d. started... 2023-04-12 02:55:34,748 INFO [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5add8dc92f700932b95abfa5a81f714d 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-04-12 02:55:34,751 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-12 02:55:34,751 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-12 02:55:34,753 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-12 02:55:34,753 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testInvalidSplitKey' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 02:55:34,753 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-12 02:55:34,753 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testInvalidSplitKey' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 02:55:34,766 INFO [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=19 (bloomFilter=true), to=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testInvalidSplitKey/5add8dc92f700932b95abfa5a81f714d/.tmp/cf1/0417b231dbbf483eb8332c94d8ffdcb4 2023-04-12 02:55:34,794 INFO [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=19 (bloomFilter=true), to=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testInvalidSplitKey/5add8dc92f700932b95abfa5a81f714d/.tmp/cf2/79d3f4e6c7594df9b9c2aed9c5d983b8 2023-04-12 02:55:34,799 DEBUG [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testInvalidSplitKey/5add8dc92f700932b95abfa5a81f714d/.tmp/cf1/0417b231dbbf483eb8332c94d8ffdcb4 as hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testInvalidSplitKey/5add8dc92f700932b95abfa5a81f714d/cf1/0417b231dbbf483eb8332c94d8ffdcb4 2023-04-12 02:55:34,804 INFO [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testInvalidSplitKey/5add8dc92f700932b95abfa5a81f714d/cf1/0417b231dbbf483eb8332c94d8ffdcb4, entries=10, sequenceid=19, filesize=5.0 K 2023-04-12 02:55:34,805 DEBUG [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testInvalidSplitKey/5add8dc92f700932b95abfa5a81f714d/.tmp/cf2/79d3f4e6c7594df9b9c2aed9c5d983b8 as hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testInvalidSplitKey/5add8dc92f700932b95abfa5a81f714d/cf2/79d3f4e6c7594df9b9c2aed9c5d983b8 2023-04-12 02:55:34,810 INFO [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testInvalidSplitKey/5add8dc92f700932b95abfa5a81f714d/cf2/79d3f4e6c7594df9b9c2aed9c5d983b8, entries=10, sequenceid=19, filesize=5.0 K 2023-04-12 02:55:34,811 INFO [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for 5add8dc92f700932b95abfa5a81f714d in 63ms, sequenceid=19, compaction requested=false 2023-04-12 02:55:34,811 DEBUG [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5add8dc92f700932b95abfa5a81f714d: 2023-04-12 02:55:34,811 DEBUG [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testInvalidSplitKey,,1681268133998.5add8dc92f700932b95abfa5a81f714d. 2023-04-12 02:55:34,811 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 02:55:34,811 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 02:55:34,811 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:34,812 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testInvalidSplitKey' locally acquired 2023-04-12 02:55:34,812 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34579,1681268031719' joining acquired barrier for procedure (testInvalidSplitKey) in zk 2023-04-12 02:55:34,814 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:34,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:34,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 02:55:34,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 02:55:34,814 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-12 02:55:34,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 02:55:34,814 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-12 02:55:34,815 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testInvalidSplitKey' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 02:55:34,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 02:55:34,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testInvalidSplitKey 2023-04-12 02:55:34,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:34,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:34,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:34,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 02:55:34,817 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34579,1681268031719' joining acquired barrier for procedure 'testInvalidSplitKey' on coordinator 2023-04-12 02:55:34,817 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@50b9e05[Count = 0] remaining members to acquire global barrier 2023-04-12 02:55:34,817 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testInvalidSplitKey' starting 'in-barrier' execution. 2023-04-12 02:55:34,817 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-12 02:55:34,818 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-12 02:55:34,818 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-12 02:55:34,818 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-12 02:55:34,818 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-12 02:55:34,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-12 02:55:34,818 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-12 02:55:34,819 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testInvalidSplitKey' received 'reached' from coordinator. 2023-04-12 02:55:34,819 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testInvalidSplitKey' locally completed 2023-04-12 02:55:34,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-12 02:55:34,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-12 02:55:34,819 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testInvalidSplitKey' received 'reached' from coordinator. 2023-04-12 02:55:34,819 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testInvalidSplitKey' locally completed 2023-04-12 02:55:34,819 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testInvalidSplitKey' completed for member 'jenkins-hbase4.apache.org,43091,1681268031804' in zk 2023-04-12 02:55:34,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-12 02:55:34,819 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testInvalidSplitKey' completed for member 'jenkins-hbase4.apache.org,44679,1681268031772' in zk 2023-04-12 02:55:34,819 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testInvalidSplitKey' received 'reached' from coordinator. 2023-04-12 02:55:34,819 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testInvalidSplitKey' locally completed 2023-04-12 02:55:34,819 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testInvalidSplitKey' completed for member 'jenkins-hbase4.apache.org,34579,1681268031719' in zk 2023-04-12 02:55:34,819 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:34,819 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 02:55:34,822 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testInvalidSplitKey' has notified controller of completion 2023-04-12 02:55:34,822 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:55:34,822 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testInvalidSplitKey' completed. 2023-04-12 02:55:34,823 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testInvalidSplitKey' has notified controller of completion 2023-04-12 02:55:34,823 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:55:34,823 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testInvalidSplitKey' completed. 2023-04-12 02:55:34,823 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testInvalidSplitKey' has notified controller of completion 2023-04-12 02:55:34,823 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:55:34,823 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testInvalidSplitKey' completed. 2023-04-12 02:55:34,823 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:34,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:34,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 02:55:34,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 02:55:34,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 02:55:34,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 02:55:34,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testInvalidSplitKey 2023-04-12 02:55:34,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:34,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:34,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:34,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 02:55:34,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testInvalidSplitKey 2023-04-12 02:55:34,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:34,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:34,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:34,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testInvalidSplitKey' member 'jenkins-hbase4.apache.org,34579,1681268031719': 2023-04-12 02:55:34,826 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34579,1681268031719' released barrier for procedure'testInvalidSplitKey', counting down latch. Waiting for 0 more 2023-04-12 02:55:34,826 INFO [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testInvalidSplitKey' execution completed 2023-04-12 02:55:34,826 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 02:55:34,826 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 02:55:34,826 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testInvalidSplitKey 2023-04-12 02:55:34,826 INFO [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testInvalidSplitKeyincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 02:55:34,828 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-12 02:55:34,828 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:55:34,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-12 02:55:34,828 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-12 02:55:34,828 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-12 02:55:34,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-12 02:55:34,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-12 02:55:34,828 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:55:34,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-12 02:55:34,828 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-12 02:55:34,828 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:55:34,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-12 02:55:34,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-12 02:55:34,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:55:34,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:55:34,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-12 02:55:34,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 02:55:34,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 02:55:34,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:55:34,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:55:34,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:55:34,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:55:34,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-12 02:55:34,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-12 02:55:34,830 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:34,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-12 02:55:34,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 02:55:34,830 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:34,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testInvalidSplitKey 2023-04-12 02:55:34,831 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:34,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 02:55:34,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testInvalidSplitKey 2023-04-12 02:55:34,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:34,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:34,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:34,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 02:55:34,832 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:34,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testInvalidSplitKey 2023-04-12 02:55:34,833 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:34,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:34,833 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:34,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:34,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:34,837 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:55:34,837 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:55:34,837 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:55:34,837 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:55:34,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:55:34,837 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:55:34,837 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:55:34,837 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:55:34,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:55:34,837 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:34,837 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-04-12 02:55:34,837 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:34,837 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:34,837 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:55:34,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:55:34,837 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:55:34,837 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:55:34,837 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testInvalidSplitKey' 2023-04-12 02:55:34,838 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 02:55:34,838 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:55:34,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:55:34,837 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-04-12 02:55:34,838 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:34,838 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:55:34,838 DEBUG [Listener at localhost/46667] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testInvalidSplitKey'' to complete. (max 20000 ms per retry) 2023-04-12 02:55:34,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:55:34,838 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-12 02:55:34,838 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:34,838 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:34,838 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-12 02:55:34,838 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-12 02:55:34,838 DEBUG [Listener at localhost/46667] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 02:55:34,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:55:34,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:55:34,939 DEBUG [Listener at localhost/46667] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 02:55:34,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 02:55:34,957 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 02:55:34,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] procedure.ProcedureCoordinator(143): Procedure testInvalidSplitKey was in running list but was completed. Accepting new attempt. 2023-04-12 02:55:34,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] procedure.ProcedureCoordinator(165): Submitting procedure testInvalidSplitKey 2023-04-12 02:55:34,960 INFO [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testInvalidSplitKey' 2023-04-12 02:55:34,960 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:55:34,960 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testInvalidSplitKey' starting 'acquire' 2023-04-12 02:55:34,960 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testInvalidSplitKey', kicking off acquire phase on members. 2023-04-12 02:55:34,961 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-12 02:55:34,961 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-04-12 02:55:34,963 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:55:34,963 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:34,963 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:55:34,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:55:34,963 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:55:34,963 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:55:34,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:55:34,964 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:55:34,964 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:34,964 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 02:55:34,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-04-12 02:55:34,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-04-12 02:55:34,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:55:34,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:55:34,964 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-12 02:55:34,964 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-12 02:55:34,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-04-12 02:55:34,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 02:55:34,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-04-12 02:55:34,965 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testInvalidSplitKey 2023-04-12 02:55:34,965 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-12 02:55:34,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 02:55:34,965 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testInvalidSplitKey is still present, removing 2023-04-12 02:55:34,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-04-12 02:55:34,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 02:55:34,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-04-12 02:55:34,965 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testInvalidSplitKey 2023-04-12 02:55:34,965 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testInvalidSplitKey 2023-04-12 02:55:34,966 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testInvalidSplitKey 2023-04-12 02:55:34,966 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testInvalidSplitKey' with timeout 60000ms 2023-04-12 02:55:34,966 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:55:34,966 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testInvalidSplitKey' starting 'acquire' stage 2023-04-12 02:55:34,966 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testInvalidSplitKey' locally acquired 2023-04-12 02:55:34,966 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43091,1681268031804' joining acquired barrier for procedure (testInvalidSplitKey) in zk 2023-04-12 02:55:34,967 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testInvalidSplitKey is still present, removing 2023-04-12 02:55:34,967 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testInvalidSplitKey 2023-04-12 02:55:34,967 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testInvalidSplitKey' with timeout 60000ms 2023-04-12 02:55:34,967 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:55:34,967 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testInvalidSplitKey' starting 'acquire' stage 2023-04-12 02:55:34,967 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testInvalidSplitKey' locally acquired 2023-04-12 02:55:34,967 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44679,1681268031772' joining acquired barrier for procedure (testInvalidSplitKey) in zk 2023-04-12 02:55:34,967 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testInvalidSplitKey is still present, removing 2023-04-12 02:55:34,967 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testInvalidSplitKey 2023-04-12 02:55:34,967 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testInvalidSplitKey' with timeout 60000ms 2023-04-12 02:55:34,967 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:55:34,967 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testInvalidSplitKey' starting 'acquire' stage 2023-04-12 02:55:34,967 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 02:55:34,967 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 02:55:34,968 DEBUG [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testInvalidSplitKey,,1681268133998.5add8dc92f700932b95abfa5a81f714d. 2023-04-12 02:55:34,968 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-12 02:55:34,968 DEBUG [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testInvalidSplitKey,,1681268133998.5add8dc92f700932b95abfa5a81f714d. started... 2023-04-12 02:55:34,968 INFO [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5add8dc92f700932b95abfa5a81f714d 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-04-12 02:55:34,969 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-12 02:55:34,969 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-12 02:55:34,969 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testInvalidSplitKey' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 02:55:34,969 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-12 02:55:34,969 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testInvalidSplitKey' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 02:55:34,988 INFO [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=32 (bloomFilter=true), to=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testInvalidSplitKey/5add8dc92f700932b95abfa5a81f714d/.tmp/cf1/53389b263afe426aa1bc1b611cf318b8 2023-04-12 02:55:35,016 INFO [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=32 (bloomFilter=true), to=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testInvalidSplitKey/5add8dc92f700932b95abfa5a81f714d/.tmp/cf2/bc014d4c1b4e4357bfb0a84cd5632dd8 2023-04-12 02:55:35,023 DEBUG [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testInvalidSplitKey/5add8dc92f700932b95abfa5a81f714d/.tmp/cf1/53389b263afe426aa1bc1b611cf318b8 as hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testInvalidSplitKey/5add8dc92f700932b95abfa5a81f714d/cf1/53389b263afe426aa1bc1b611cf318b8 2023-04-12 02:55:35,028 INFO [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testInvalidSplitKey/5add8dc92f700932b95abfa5a81f714d/cf1/53389b263afe426aa1bc1b611cf318b8, entries=10, sequenceid=32, filesize=5.0 K 2023-04-12 02:55:35,028 DEBUG [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testInvalidSplitKey/5add8dc92f700932b95abfa5a81f714d/.tmp/cf2/bc014d4c1b4e4357bfb0a84cd5632dd8 as hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testInvalidSplitKey/5add8dc92f700932b95abfa5a81f714d/cf2/bc014d4c1b4e4357bfb0a84cd5632dd8 2023-04-12 02:55:35,032 INFO [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testInvalidSplitKey/5add8dc92f700932b95abfa5a81f714d/cf2/bc014d4c1b4e4357bfb0a84cd5632dd8, entries=10, sequenceid=32, filesize=5.0 K 2023-04-12 02:55:35,033 INFO [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for 5add8dc92f700932b95abfa5a81f714d in 65ms, sequenceid=32, compaction requested=true 2023-04-12 02:55:35,033 DEBUG [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5add8dc92f700932b95abfa5a81f714d: 2023-04-12 02:55:35,034 DEBUG [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testInvalidSplitKey,,1681268133998.5add8dc92f700932b95abfa5a81f714d. 2023-04-12 02:55:35,034 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 02:55:35,034 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 02:55:35,034 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:35,034 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testInvalidSplitKey' locally acquired 2023-04-12 02:55:35,034 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34579,1681268031719' joining acquired barrier for procedure (testInvalidSplitKey) in zk 2023-04-12 02:55:35,035 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:35,035 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-12 02:55:35,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:35,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 02:55:35,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 02:55:35,036 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-12 02:55:35,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 02:55:35,036 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testInvalidSplitKey' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 02:55:35,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 02:55:35,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testInvalidSplitKey 2023-04-12 02:55:35,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:35,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:35,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:35,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 02:55:35,038 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34579,1681268031719' joining acquired barrier for procedure 'testInvalidSplitKey' on coordinator 2023-04-12 02:55:35,038 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@183dc08c[Count = 0] remaining members to acquire global barrier 2023-04-12 02:55:35,038 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testInvalidSplitKey' starting 'in-barrier' execution. 2023-04-12 02:55:35,038 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-12 02:55:35,040 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-12 02:55:35,040 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-12 02:55:35,040 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-12 02:55:35,040 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-12 02:55:35,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-12 02:55:35,040 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-12 02:55:35,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-12 02:55:35,040 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testInvalidSplitKey' received 'reached' from coordinator. 2023-04-12 02:55:35,041 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testInvalidSplitKey' received 'reached' from coordinator. 2023-04-12 02:55:35,040 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-12 02:55:35,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-12 02:55:35,041 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testInvalidSplitKey' locally completed 2023-04-12 02:55:35,041 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testInvalidSplitKey' locally completed 2023-04-12 02:55:35,041 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testInvalidSplitKey' completed for member 'jenkins-hbase4.apache.org,34579,1681268031719' in zk 2023-04-12 02:55:35,041 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testInvalidSplitKey' completed for member 'jenkins-hbase4.apache.org,43091,1681268031804' in zk 2023-04-12 02:55:35,041 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testInvalidSplitKey' received 'reached' from coordinator. 2023-04-12 02:55:35,041 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testInvalidSplitKey' locally completed 2023-04-12 02:55:35,041 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testInvalidSplitKey' completed for member 'jenkins-hbase4.apache.org,44679,1681268031772' in zk 2023-04-12 02:55:35,042 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:35,042 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 02:55:35,043 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testInvalidSplitKey' has notified controller of completion 2023-04-12 02:55:35,043 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:35,043 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testInvalidSplitKey' has notified controller of completion 2023-04-12 02:55:35,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:35,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 02:55:35,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 02:55:35,043 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:55:35,043 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:55:35,044 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testInvalidSplitKey' completed. 2023-04-12 02:55:35,044 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testInvalidSplitKey' completed. 2023-04-12 02:55:35,044 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testInvalidSplitKey' has notified controller of completion 2023-04-12 02:55:35,044 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:55:35,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 02:55:35,044 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testInvalidSplitKey' completed. 2023-04-12 02:55:35,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 02:55:35,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testInvalidSplitKey 2023-04-12 02:55:35,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:35,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:35,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:35,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 02:55:35,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testInvalidSplitKey 2023-04-12 02:55:35,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:35,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:35,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:35,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testInvalidSplitKey' member 'jenkins-hbase4.apache.org,34579,1681268031719': 2023-04-12 02:55:35,048 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34579,1681268031719' released barrier for procedure'testInvalidSplitKey', counting down latch. Waiting for 0 more 2023-04-12 02:55:35,048 INFO [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testInvalidSplitKey' execution completed 2023-04-12 02:55:35,048 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 02:55:35,048 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 02:55:35,048 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testInvalidSplitKey 2023-04-12 02:55:35,048 INFO [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testInvalidSplitKeyincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 02:55:35,051 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-12 02:55:35,051 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-12 02:55:35,051 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-12 02:55:35,051 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:55:35,051 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-12 02:55:35,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-12 02:55:35,051 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-12 02:55:35,051 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:55:35,051 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-12 02:55:35,051 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-12 02:55:35,051 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:55:35,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-12 02:55:35,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-12 02:55:35,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-12 02:55:35,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 02:55:35,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 02:55:35,052 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:55:35,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:55:35,052 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:55:35,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:55:35,052 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:55:35,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:55:35,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-12 02:55:35,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-12 02:55:35,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-12 02:55:35,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 02:55:35,054 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:35,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testInvalidSplitKey 2023-04-12 02:55:35,058 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:35,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 02:55:35,058 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:35,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testInvalidSplitKey 2023-04-12 02:55:35,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:35,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:35,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:35,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 02:55:35,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testInvalidSplitKey 2023-04-12 02:55:35,061 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:35,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:35,061 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:35,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:35,061 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:35,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:35,069 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:55:35,069 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:55:35,069 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:55:35,069 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:35,069 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testInvalidSplitKey' 2023-04-12 02:55:35,069 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:55:35,069 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:55:35,069 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:55:35,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:55:35,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:55:35,069 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:55:35,069 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 02:55:35,069 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:55:35,070 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:55:35,069 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-04-12 02:55:35,070 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:35,070 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:35,070 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-04-12 02:55:35,070 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:35,070 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-12 02:55:35,070 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:35,070 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:35,070 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-12 02:55:35,070 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-12 02:55:35,070 DEBUG [Listener at localhost/46667] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testInvalidSplitKey'' to complete. (max 20000 ms per retry) 2023-04-12 02:55:35,070 DEBUG [Listener at localhost/46667] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 02:55:35,070 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:55:35,070 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:55:35,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:55:35,070 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:55:35,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:55:35,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:55:35,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:55:35,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:55:35,170 DEBUG [Listener at localhost/46667] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 02:55:35,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 02:55:35,186 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 02:55:35,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] procedure.ProcedureCoordinator(143): Procedure testInvalidSplitKey was in running list but was completed. Accepting new attempt. 2023-04-12 02:55:35,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] procedure.ProcedureCoordinator(165): Submitting procedure testInvalidSplitKey 2023-04-12 02:55:35,188 INFO [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testInvalidSplitKey' 2023-04-12 02:55:35,188 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:55:35,188 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testInvalidSplitKey' starting 'acquire' 2023-04-12 02:55:35,188 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testInvalidSplitKey', kicking off acquire phase on members. 2023-04-12 02:55:35,188 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-12 02:55:35,188 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-04-12 02:55:35,190 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:55:35,190 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:35,190 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:55:35,190 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:55:35,190 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:55:35,190 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:55:35,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:55:35,190 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:55:35,190 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:35,190 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 02:55:35,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:55:35,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:55:35,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-04-12 02:55:35,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-04-12 02:55:35,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-04-12 02:55:35,191 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-12 02:55:35,191 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-12 02:55:35,191 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-12 02:55:35,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 02:55:35,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-04-12 02:55:35,191 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testInvalidSplitKey 2023-04-12 02:55:35,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 02:55:35,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-04-12 02:55:35,191 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testInvalidSplitKey 2023-04-12 02:55:35,191 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testInvalidSplitKey is still present, removing 2023-04-12 02:55:35,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 02:55:35,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-04-12 02:55:35,192 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testInvalidSplitKey 2023-04-12 02:55:35,192 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testInvalidSplitKey is still present, removing 2023-04-12 02:55:35,192 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testInvalidSplitKey 2023-04-12 02:55:35,192 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testInvalidSplitKey 2023-04-12 02:55:35,192 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testInvalidSplitKey' with timeout 60000ms 2023-04-12 02:55:35,192 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:55:35,192 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testInvalidSplitKey' with timeout 60000ms 2023-04-12 02:55:35,192 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:55:35,192 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testInvalidSplitKey' starting 'acquire' stage 2023-04-12 02:55:35,192 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testInvalidSplitKey' starting 'acquire' stage 2023-04-12 02:55:35,192 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testInvalidSplitKey' locally acquired 2023-04-12 02:55:35,192 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testInvalidSplitKey' locally acquired 2023-04-12 02:55:35,192 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44679,1681268031772' joining acquired barrier for procedure (testInvalidSplitKey) in zk 2023-04-12 02:55:35,192 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testInvalidSplitKey is still present, removing 2023-04-12 02:55:35,192 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testInvalidSplitKey 2023-04-12 02:55:35,192 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43091,1681268031804' joining acquired barrier for procedure (testInvalidSplitKey) in zk 2023-04-12 02:55:35,192 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testInvalidSplitKey' with timeout 60000ms 2023-04-12 02:55:35,192 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:55:35,192 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testInvalidSplitKey' starting 'acquire' stage 2023-04-12 02:55:35,193 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 02:55:35,193 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 02:55:35,193 DEBUG [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testInvalidSplitKey,,1681268133998.5add8dc92f700932b95abfa5a81f714d. 2023-04-12 02:55:35,193 DEBUG [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testInvalidSplitKey,,1681268133998.5add8dc92f700932b95abfa5a81f714d. started... 2023-04-12 02:55:35,193 INFO [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5add8dc92f700932b95abfa5a81f714d 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-04-12 02:55:35,194 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-12 02:55:35,194 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-12 02:55:35,195 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-12 02:55:35,195 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-12 02:55:35,195 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testInvalidSplitKey' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 02:55:35,195 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testInvalidSplitKey' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 02:55:35,207 INFO [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=45 (bloomFilter=true), to=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testInvalidSplitKey/5add8dc92f700932b95abfa5a81f714d/.tmp/cf1/fcd93194ff764d89b7767e7ac2433220 2023-04-12 02:55:35,234 INFO [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=45 (bloomFilter=true), to=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testInvalidSplitKey/5add8dc92f700932b95abfa5a81f714d/.tmp/cf2/e2288421480743338fec14aaced87369 2023-04-12 02:55:35,240 DEBUG [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testInvalidSplitKey/5add8dc92f700932b95abfa5a81f714d/.tmp/cf1/fcd93194ff764d89b7767e7ac2433220 as hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testInvalidSplitKey/5add8dc92f700932b95abfa5a81f714d/cf1/fcd93194ff764d89b7767e7ac2433220 2023-04-12 02:55:35,245 INFO [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testInvalidSplitKey/5add8dc92f700932b95abfa5a81f714d/cf1/fcd93194ff764d89b7767e7ac2433220, entries=10, sequenceid=45, filesize=5.0 K 2023-04-12 02:55:35,246 DEBUG [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testInvalidSplitKey/5add8dc92f700932b95abfa5a81f714d/.tmp/cf2/e2288421480743338fec14aaced87369 as hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testInvalidSplitKey/5add8dc92f700932b95abfa5a81f714d/cf2/e2288421480743338fec14aaced87369 2023-04-12 02:55:35,250 INFO [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testInvalidSplitKey/5add8dc92f700932b95abfa5a81f714d/cf2/e2288421480743338fec14aaced87369, entries=10, sequenceid=45, filesize=5.0 K 2023-04-12 02:55:35,251 INFO [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for 5add8dc92f700932b95abfa5a81f714d in 58ms, sequenceid=45, compaction requested=true 2023-04-12 02:55:35,251 DEBUG [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5add8dc92f700932b95abfa5a81f714d: 2023-04-12 02:55:35,251 DEBUG [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testInvalidSplitKey,,1681268133998.5add8dc92f700932b95abfa5a81f714d. 2023-04-12 02:55:35,251 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 02:55:35,251 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 02:55:35,251 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:35,251 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testInvalidSplitKey' locally acquired 2023-04-12 02:55:35,251 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34579,1681268031719' joining acquired barrier for procedure (testInvalidSplitKey) in zk 2023-04-12 02:55:35,253 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:35,253 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-12 02:55:35,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:35,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 02:55:35,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 02:55:35,254 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-12 02:55:35,254 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testInvalidSplitKey' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 02:55:35,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 02:55:35,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 02:55:35,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testInvalidSplitKey 2023-04-12 02:55:35,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:35,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:35,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:35,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 02:55:35,256 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34579,1681268031719' joining acquired barrier for procedure 'testInvalidSplitKey' on coordinator 2023-04-12 02:55:35,256 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@12f944ed[Count = 0] remaining members to acquire global barrier 2023-04-12 02:55:35,256 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testInvalidSplitKey' starting 'in-barrier' execution. 2023-04-12 02:55:35,256 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-12 02:55:35,258 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-12 02:55:35,258 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-12 02:55:35,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-12 02:55:35,258 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-12 02:55:35,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-12 02:55:35,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-12 02:55:35,258 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testInvalidSplitKey' received 'reached' from coordinator. 2023-04-12 02:55:35,258 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testInvalidSplitKey' locally completed 2023-04-12 02:55:35,258 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testInvalidSplitKey' completed for member 'jenkins-hbase4.apache.org,44679,1681268031772' in zk 2023-04-12 02:55:35,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-12 02:55:35,258 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:35,258 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 02:55:35,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-12 02:55:35,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-12 02:55:35,259 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testInvalidSplitKey' received 'reached' from coordinator. 2023-04-12 02:55:35,259 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testInvalidSplitKey' received 'reached' from coordinator. 2023-04-12 02:55:35,259 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testInvalidSplitKey' locally completed 2023-04-12 02:55:35,259 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testInvalidSplitKey' completed for member 'jenkins-hbase4.apache.org,43091,1681268031804' in zk 2023-04-12 02:55:35,259 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testInvalidSplitKey' locally completed 2023-04-12 02:55:35,259 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testInvalidSplitKey' completed for member 'jenkins-hbase4.apache.org,34579,1681268031719' in zk 2023-04-12 02:55:35,260 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testInvalidSplitKey' has notified controller of completion 2023-04-12 02:55:35,260 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:55:35,260 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testInvalidSplitKey' completed. 2023-04-12 02:55:35,262 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testInvalidSplitKey' has notified controller of completion 2023-04-12 02:55:35,262 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:55:35,262 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:35,262 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testInvalidSplitKey' completed. 2023-04-12 02:55:35,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:35,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 02:55:35,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 02:55:35,263 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testInvalidSplitKey' has notified controller of completion 2023-04-12 02:55:35,263 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:55:35,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 02:55:35,263 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testInvalidSplitKey' completed. 2023-04-12 02:55:35,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 02:55:35,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testInvalidSplitKey 2023-04-12 02:55:35,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:35,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:35,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:35,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 02:55:35,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testInvalidSplitKey 2023-04-12 02:55:35,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:35,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:35,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:35,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testInvalidSplitKey' member 'jenkins-hbase4.apache.org,34579,1681268031719': 2023-04-12 02:55:35,266 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34579,1681268031719' released barrier for procedure'testInvalidSplitKey', counting down latch. Waiting for 0 more 2023-04-12 02:55:35,266 INFO [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testInvalidSplitKey' execution completed 2023-04-12 02:55:35,266 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 02:55:35,266 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 02:55:35,266 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testInvalidSplitKey 2023-04-12 02:55:35,266 INFO [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testInvalidSplitKeyincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 02:55:35,269 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-12 02:55:35,269 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-12 02:55:35,269 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-12 02:55:35,269 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-12 02:55:35,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-12 02:55:35,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 02:55:35,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 02:55:35,269 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:55:35,269 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-12 02:55:35,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-12 02:55:35,269 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:55:35,269 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-12 02:55:35,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-12 02:55:35,270 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-12 02:55:35,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-12 02:55:35,269 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:55:35,270 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:35,270 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:55:35,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:55:35,270 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:55:35,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:55:35,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 02:55:35,270 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:55:35,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:55:35,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-12 02:55:35,270 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:35,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-12 02:55:35,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-12 02:55:35,271 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:35,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testInvalidSplitKey 2023-04-12 02:55:35,272 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:35,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 02:55:35,272 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:35,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testInvalidSplitKey 2023-04-12 02:55:35,272 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:35,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:35,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:35,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:35,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 02:55:35,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testInvalidSplitKey 2023-04-12 02:55:35,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:35,276 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:35,276 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-04-12 02:55:35,276 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:35,276 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:55:35,276 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:35,276 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:55:35,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:55:35,276 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:55:35,276 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:55:35,276 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-04-12 02:55:35,276 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:55:35,276 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:55:35,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:35,276 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:35,276 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-12 02:55:35,276 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:35,276 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:35,276 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-12 02:55:35,276 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:55:35,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:55:35,276 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testInvalidSplitKey' 2023-04-12 02:55:35,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:35,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:55:35,276 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:55:35,277 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:55:35,277 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:55:35,277 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 02:55:35,276 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-12 02:55:35,277 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:55:35,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:55:35,277 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:55:35,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:55:35,277 DEBUG [Listener at localhost/46667] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testInvalidSplitKey'' to complete. (max 20000 ms per retry) 2023-04-12 02:55:35,277 DEBUG [Listener at localhost/46667] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 02:55:35,277 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:55:35,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:55:35,280 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/testInvalidSplitKey because node does not exist (not an error) 2023-04-12 02:55:35,377 DEBUG [Listener at localhost/46667] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 02:55:35,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 02:55:35,395 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 02:55:35,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] procedure.ProcedureCoordinator(143): Procedure testInvalidSplitKey was in running list but was completed. Accepting new attempt. 2023-04-12 02:55:35,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] procedure.ProcedureCoordinator(165): Submitting procedure testInvalidSplitKey 2023-04-12 02:55:35,397 INFO [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testInvalidSplitKey' 2023-04-12 02:55:35,397 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:55:35,397 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testInvalidSplitKey' starting 'acquire' 2023-04-12 02:55:35,397 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testInvalidSplitKey', kicking off acquire phase on members. 2023-04-12 02:55:35,398 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-12 02:55:35,398 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-04-12 02:55:35,400 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:55:35,400 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:55:35,400 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:55:35,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:55:35,400 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:55:35,400 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:55:35,400 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:35,400 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:55:35,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:55:35,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:55:35,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-04-12 02:55:35,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-04-12 02:55:35,401 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:35,401 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 02:55:35,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-04-12 02:55:35,401 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-12 02:55:35,401 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-12 02:55:35,401 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-12 02:55:35,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 02:55:35,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-04-12 02:55:35,401 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testInvalidSplitKey 2023-04-12 02:55:35,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 02:55:35,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-04-12 02:55:35,402 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testInvalidSplitKey 2023-04-12 02:55:35,402 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testInvalidSplitKey is still present, removing 2023-04-12 02:55:35,402 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testInvalidSplitKey 2023-04-12 02:55:35,402 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testInvalidSplitKey is still present, removing 2023-04-12 02:55:35,402 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testInvalidSplitKey' with timeout 60000ms 2023-04-12 02:55:35,402 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:55:35,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 02:55:35,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-04-12 02:55:35,402 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testInvalidSplitKey 2023-04-12 02:55:35,402 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testInvalidSplitKey' starting 'acquire' stage 2023-04-12 02:55:35,402 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testInvalidSplitKey 2023-04-12 02:55:35,402 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testInvalidSplitKey is still present, removing 2023-04-12 02:55:35,402 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testInvalidSplitKey' with timeout 60000ms 2023-04-12 02:55:35,403 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:55:35,403 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testInvalidSplitKey' starting 'acquire' stage 2023-04-12 02:55:35,402 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testInvalidSplitKey' locally acquired 2023-04-12 02:55:35,402 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testInvalidSplitKey 2023-04-12 02:55:35,403 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43091,1681268031804' joining acquired barrier for procedure (testInvalidSplitKey) in zk 2023-04-12 02:55:35,403 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testInvalidSplitKey' with timeout 60000ms 2023-04-12 02:55:35,403 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:55:35,403 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testInvalidSplitKey' starting 'acquire' stage 2023-04-12 02:55:35,403 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testInvalidSplitKey' locally acquired 2023-04-12 02:55:35,403 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44679,1681268031772' joining acquired barrier for procedure (testInvalidSplitKey) in zk 2023-04-12 02:55:35,404 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 02:55:35,404 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 02:55:35,404 DEBUG [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testInvalidSplitKey,,1681268133998.5add8dc92f700932b95abfa5a81f714d. 2023-04-12 02:55:35,404 DEBUG [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testInvalidSplitKey,,1681268133998.5add8dc92f700932b95abfa5a81f714d. started... 2023-04-12 02:55:35,404 INFO [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5add8dc92f700932b95abfa5a81f714d 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-04-12 02:55:35,405 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-12 02:55:35,406 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-12 02:55:35,406 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testInvalidSplitKey' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 02:55:35,406 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-12 02:55:35,406 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-12 02:55:35,406 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testInvalidSplitKey' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 02:55:35,423 INFO [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=58 (bloomFilter=true), to=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testInvalidSplitKey/5add8dc92f700932b95abfa5a81f714d/.tmp/cf1/58c01e644f78443db3caaba4ffc33a47 2023-04-12 02:55:35,445 INFO [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=58 (bloomFilter=true), to=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testInvalidSplitKey/5add8dc92f700932b95abfa5a81f714d/.tmp/cf2/c882be8d0ad34abf9c16dce48a03abeb 2023-04-12 02:55:35,450 DEBUG [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testInvalidSplitKey/5add8dc92f700932b95abfa5a81f714d/.tmp/cf1/58c01e644f78443db3caaba4ffc33a47 as hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testInvalidSplitKey/5add8dc92f700932b95abfa5a81f714d/cf1/58c01e644f78443db3caaba4ffc33a47 2023-04-12 02:55:35,453 INFO [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testInvalidSplitKey/5add8dc92f700932b95abfa5a81f714d/cf1/58c01e644f78443db3caaba4ffc33a47, entries=10, sequenceid=58, filesize=5.0 K 2023-04-12 02:55:35,454 DEBUG [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testInvalidSplitKey/5add8dc92f700932b95abfa5a81f714d/.tmp/cf2/c882be8d0ad34abf9c16dce48a03abeb as hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testInvalidSplitKey/5add8dc92f700932b95abfa5a81f714d/cf2/c882be8d0ad34abf9c16dce48a03abeb 2023-04-12 02:55:35,459 INFO [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testInvalidSplitKey/5add8dc92f700932b95abfa5a81f714d/cf2/c882be8d0ad34abf9c16dce48a03abeb, entries=10, sequenceid=58, filesize=5.0 K 2023-04-12 02:55:35,460 INFO [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for 5add8dc92f700932b95abfa5a81f714d in 56ms, sequenceid=58, compaction requested=true 2023-04-12 02:55:35,460 DEBUG [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5add8dc92f700932b95abfa5a81f714d: 2023-04-12 02:55:35,460 DEBUG [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testInvalidSplitKey,,1681268133998.5add8dc92f700932b95abfa5a81f714d. 2023-04-12 02:55:35,460 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 02:55:35,460 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 02:55:35,460 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:35,460 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testInvalidSplitKey' locally acquired 2023-04-12 02:55:35,460 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34579,1681268031719' joining acquired barrier for procedure (testInvalidSplitKey) in zk 2023-04-12 02:55:35,462 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:35,462 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-12 02:55:35,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:35,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 02:55:35,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 02:55:35,463 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-12 02:55:35,463 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testInvalidSplitKey' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 02:55:35,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 02:55:35,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 02:55:35,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testInvalidSplitKey 2023-04-12 02:55:35,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:35,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:35,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:35,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 02:55:35,465 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34579,1681268031719' joining acquired barrier for procedure 'testInvalidSplitKey' on coordinator 2023-04-12 02:55:35,465 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1835cbf2[Count = 0] remaining members to acquire global barrier 2023-04-12 02:55:35,465 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testInvalidSplitKey' starting 'in-barrier' execution. 2023-04-12 02:55:35,465 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-12 02:55:35,467 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-12 02:55:35,467 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-12 02:55:35,467 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-12 02:55:35,467 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-12 02:55:35,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-12 02:55:35,467 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-12 02:55:35,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-12 02:55:35,468 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-12 02:55:35,468 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testInvalidSplitKey' received 'reached' from coordinator. 2023-04-12 02:55:35,468 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testInvalidSplitKey' locally completed 2023-04-12 02:55:35,468 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testInvalidSplitKey' received 'reached' from coordinator. 2023-04-12 02:55:35,468 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testInvalidSplitKey' completed for member 'jenkins-hbase4.apache.org,34579,1681268031719' in zk 2023-04-12 02:55:35,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-12 02:55:35,468 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testInvalidSplitKey' received 'reached' from coordinator. 2023-04-12 02:55:35,468 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testInvalidSplitKey' locally completed 2023-04-12 02:55:35,468 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testInvalidSplitKey' completed for member 'jenkins-hbase4.apache.org,43091,1681268031804' in zk 2023-04-12 02:55:35,468 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testInvalidSplitKey' locally completed 2023-04-12 02:55:35,468 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testInvalidSplitKey' completed for member 'jenkins-hbase4.apache.org,44679,1681268031772' in zk 2023-04-12 02:55:35,469 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:35,469 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 02:55:35,472 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:35,472 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testInvalidSplitKey' has notified controller of completion 2023-04-12 02:55:35,472 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testInvalidSplitKey' has notified controller of completion 2023-04-12 02:55:35,472 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:55:35,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:35,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 02:55:35,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 02:55:35,472 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testInvalidSplitKey' completed. 2023-04-12 02:55:35,472 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:55:35,472 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testInvalidSplitKey' completed. 2023-04-12 02:55:35,476 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testInvalidSplitKey' has notified controller of completion 2023-04-12 02:55:35,476 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:55:35,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 02:55:35,476 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testInvalidSplitKey' completed. 2023-04-12 02:55:35,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 02:55:35,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testInvalidSplitKey 2023-04-12 02:55:35,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:35,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:35,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:35,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 02:55:35,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testInvalidSplitKey 2023-04-12 02:55:35,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:35,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:35,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:35,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testInvalidSplitKey' member 'jenkins-hbase4.apache.org,34579,1681268031719': 2023-04-12 02:55:35,479 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34579,1681268031719' released barrier for procedure'testInvalidSplitKey', counting down latch. Waiting for 0 more 2023-04-12 02:55:35,479 INFO [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testInvalidSplitKey' execution completed 2023-04-12 02:55:35,479 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 02:55:35,479 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 02:55:35,479 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testInvalidSplitKey 2023-04-12 02:55:35,479 INFO [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testInvalidSplitKeyincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 02:55:35,482 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-12 02:55:35,482 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-12 02:55:35,482 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-12 02:55:35,482 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:55:35,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-12 02:55:35,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 02:55:35,483 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-12 02:55:35,482 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-12 02:55:35,483 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-12 02:55:35,482 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-12 02:55:35,482 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:55:35,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-12 02:55:35,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-12 02:55:35,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-12 02:55:35,483 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:55:35,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 02:55:35,483 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:55:35,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:55:35,483 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:55:35,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:55:35,483 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:55:35,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:55:35,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 02:55:35,483 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:35,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-12 02:55:35,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-12 02:55:35,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-12 02:55:35,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testInvalidSplitKey 2023-04-12 02:55:35,484 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:35,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 02:55:35,484 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:35,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testInvalidSplitKey 2023-04-12 02:55:35,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:35,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:35,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:35,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 02:55:35,486 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:35,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testInvalidSplitKey 2023-04-12 02:55:35,486 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:35,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:35,487 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:35,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:35,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:35,489 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:35,489 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-04-12 02:55:35,489 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:35,489 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:55:35,489 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:55:35,489 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:55:35,489 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:35,489 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:55:35,489 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:55:35,489 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testInvalidSplitKey' 2023-04-12 02:55:35,489 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 02:55:35,489 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:55:35,489 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-04-12 02:55:35,490 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:55:35,489 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:55:35,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:55:35,489 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:55:35,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:55:35,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:55:35,490 DEBUG [Listener at localhost/46667] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testInvalidSplitKey'' to complete. (max 20000 ms per retry) 2023-04-12 02:55:35,490 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:35,489 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:55:35,490 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-12 02:55:35,490 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:55:35,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:55:35,490 DEBUG [Listener at localhost/46667] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 02:55:35,490 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:35,490 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:55:35,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:55:35,490 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:55:35,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:55:35,490 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:35,490 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-12 02:55:35,490 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-12 02:55:35,590 DEBUG [Listener at localhost/46667] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 02:55:35,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 02:55:35,604 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 02:55:35,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] procedure.ProcedureCoordinator(143): Procedure testInvalidSplitKey was in running list but was completed. Accepting new attempt. 2023-04-12 02:55:35,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] procedure.ProcedureCoordinator(165): Submitting procedure testInvalidSplitKey 2023-04-12 02:55:35,606 INFO [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testInvalidSplitKey' 2023-04-12 02:55:35,606 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:55:35,606 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testInvalidSplitKey' starting 'acquire' 2023-04-12 02:55:35,606 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testInvalidSplitKey', kicking off acquire phase on members. 2023-04-12 02:55:35,606 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-12 02:55:35,606 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-04-12 02:55:35,607 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:55:35,607 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:55:35,607 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:55:35,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:55:35,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:55:35,607 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:35,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:55:35,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:55:35,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:55:35,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:55:35,608 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:35,608 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 02:55:35,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-04-12 02:55:35,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-04-12 02:55:35,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-04-12 02:55:35,609 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-12 02:55:35,609 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-12 02:55:35,609 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-12 02:55:35,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 02:55:35,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-04-12 02:55:35,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 02:55:35,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-04-12 02:55:35,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 02:55:35,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-04-12 02:55:35,609 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testInvalidSplitKey 2023-04-12 02:55:35,609 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testInvalidSplitKey 2023-04-12 02:55:35,609 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testInvalidSplitKey 2023-04-12 02:55:35,609 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testInvalidSplitKey is still present, removing 2023-04-12 02:55:35,609 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testInvalidSplitKey is still present, removing 2023-04-12 02:55:35,610 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testInvalidSplitKey 2023-04-12 02:55:35,610 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testInvalidSplitKey 2023-04-12 02:55:35,610 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testInvalidSplitKey' with timeout 60000ms 2023-04-12 02:55:35,610 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testInvalidSplitKey' with timeout 60000ms 2023-04-12 02:55:35,610 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:55:35,610 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testInvalidSplitKey' starting 'acquire' stage 2023-04-12 02:55:35,610 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testInvalidSplitKey is still present, removing 2023-04-12 02:55:35,610 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 02:55:35,610 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 02:55:35,610 DEBUG [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testInvalidSplitKey,,1681268133998.5add8dc92f700932b95abfa5a81f714d. 2023-04-12 02:55:35,610 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:55:35,610 DEBUG [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testInvalidSplitKey,,1681268133998.5add8dc92f700932b95abfa5a81f714d. started... 2023-04-12 02:55:35,610 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testInvalidSplitKey 2023-04-12 02:55:35,610 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testInvalidSplitKey' starting 'acquire' stage 2023-04-12 02:55:35,610 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testInvalidSplitKey' locally acquired 2023-04-12 02:55:35,610 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44679,1681268031772' joining acquired barrier for procedure (testInvalidSplitKey) in zk 2023-04-12 02:55:35,610 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testInvalidSplitKey' with timeout 60000ms 2023-04-12 02:55:35,610 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:55:35,610 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testInvalidSplitKey' starting 'acquire' stage 2023-04-12 02:55:35,610 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testInvalidSplitKey' locally acquired 2023-04-12 02:55:35,610 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43091,1681268031804' joining acquired barrier for procedure (testInvalidSplitKey) in zk 2023-04-12 02:55:35,610 INFO [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5add8dc92f700932b95abfa5a81f714d 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-04-12 02:55:35,612 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-12 02:55:35,613 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-12 02:55:35,613 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-12 02:55:35,613 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testInvalidSplitKey' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 02:55:35,614 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-12 02:55:35,614 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testInvalidSplitKey' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 02:55:35,643 INFO [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=71 (bloomFilter=true), to=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testInvalidSplitKey/5add8dc92f700932b95abfa5a81f714d/.tmp/cf1/a9e4df0381a9416bab6903def5777a58 2023-04-12 02:55:35,666 INFO [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=71 (bloomFilter=true), to=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testInvalidSplitKey/5add8dc92f700932b95abfa5a81f714d/.tmp/cf2/9b376f96ee5d42108c970007d7a46c72 2023-04-12 02:55:35,670 DEBUG [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testInvalidSplitKey/5add8dc92f700932b95abfa5a81f714d/.tmp/cf1/a9e4df0381a9416bab6903def5777a58 as hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testInvalidSplitKey/5add8dc92f700932b95abfa5a81f714d/cf1/a9e4df0381a9416bab6903def5777a58 2023-04-12 02:55:35,674 INFO [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testInvalidSplitKey/5add8dc92f700932b95abfa5a81f714d/cf1/a9e4df0381a9416bab6903def5777a58, entries=10, sequenceid=71, filesize=5.0 K 2023-04-12 02:55:35,675 DEBUG [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testInvalidSplitKey/5add8dc92f700932b95abfa5a81f714d/.tmp/cf2/9b376f96ee5d42108c970007d7a46c72 as hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testInvalidSplitKey/5add8dc92f700932b95abfa5a81f714d/cf2/9b376f96ee5d42108c970007d7a46c72 2023-04-12 02:55:35,679 INFO [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testInvalidSplitKey/5add8dc92f700932b95abfa5a81f714d/cf2/9b376f96ee5d42108c970007d7a46c72, entries=10, sequenceid=71, filesize=5.0 K 2023-04-12 02:55:35,680 INFO [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for 5add8dc92f700932b95abfa5a81f714d in 70ms, sequenceid=71, compaction requested=true 2023-04-12 02:55:35,680 DEBUG [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5add8dc92f700932b95abfa5a81f714d: 2023-04-12 02:55:35,680 DEBUG [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testInvalidSplitKey,,1681268133998.5add8dc92f700932b95abfa5a81f714d. 2023-04-12 02:55:35,680 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 02:55:35,680 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 02:55:35,680 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:35,680 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testInvalidSplitKey' locally acquired 2023-04-12 02:55:35,680 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34579,1681268031719' joining acquired barrier for procedure (testInvalidSplitKey) in zk 2023-04-12 02:55:35,682 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:35,682 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-12 02:55:35,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:35,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 02:55:35,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 02:55:35,682 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-12 02:55:35,683 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testInvalidSplitKey' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 02:55:35,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 02:55:35,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 02:55:35,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testInvalidSplitKey 2023-04-12 02:55:35,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:35,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:35,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:35,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 02:55:35,686 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34579,1681268031719' joining acquired barrier for procedure 'testInvalidSplitKey' on coordinator 2023-04-12 02:55:35,686 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@41979547[Count = 0] remaining members to acquire global barrier 2023-04-12 02:55:35,686 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testInvalidSplitKey' starting 'in-barrier' execution. 2023-04-12 02:55:35,686 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-12 02:55:35,687 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-12 02:55:35,687 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-12 02:55:35,687 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-12 02:55:35,687 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-12 02:55:35,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-12 02:55:35,688 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testInvalidSplitKey' received 'reached' from coordinator. 2023-04-12 02:55:35,688 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testInvalidSplitKey' locally completed 2023-04-12 02:55:35,688 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testInvalidSplitKey' completed for member 'jenkins-hbase4.apache.org,43091,1681268031804' in zk 2023-04-12 02:55:35,687 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-12 02:55:35,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-12 02:55:35,688 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-12 02:55:35,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-12 02:55:35,688 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testInvalidSplitKey' received 'reached' from coordinator. 2023-04-12 02:55:35,688 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testInvalidSplitKey' locally completed 2023-04-12 02:55:35,688 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testInvalidSplitKey' completed for member 'jenkins-hbase4.apache.org,44679,1681268031772' in zk 2023-04-12 02:55:35,688 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:35,688 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 02:55:35,689 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testInvalidSplitKey' received 'reached' from coordinator. 2023-04-12 02:55:35,689 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testInvalidSplitKey' locally completed 2023-04-12 02:55:35,689 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testInvalidSplitKey' completed for member 'jenkins-hbase4.apache.org,34579,1681268031719' in zk 2023-04-12 02:55:35,689 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testInvalidSplitKey' has notified controller of completion 2023-04-12 02:55:35,689 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:55:35,689 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testInvalidSplitKey' completed. 2023-04-12 02:55:35,691 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testInvalidSplitKey' has notified controller of completion 2023-04-12 02:55:35,691 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:55:35,691 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testInvalidSplitKey' completed. 2023-04-12 02:55:35,692 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:35,692 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testInvalidSplitKey' has notified controller of completion 2023-04-12 02:55:35,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:35,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 02:55:35,692 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:55:35,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 02:55:35,692 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testInvalidSplitKey' completed. 2023-04-12 02:55:35,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 02:55:35,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 02:55:35,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testInvalidSplitKey 2023-04-12 02:55:35,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:35,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:35,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:35,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 02:55:35,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testInvalidSplitKey 2023-04-12 02:55:35,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:35,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:35,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:35,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testInvalidSplitKey' member 'jenkins-hbase4.apache.org,34579,1681268031719': 2023-04-12 02:55:35,696 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34579,1681268031719' released barrier for procedure'testInvalidSplitKey', counting down latch. Waiting for 0 more 2023-04-12 02:55:35,696 INFO [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testInvalidSplitKey' execution completed 2023-04-12 02:55:35,696 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 02:55:35,696 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 02:55:35,696 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testInvalidSplitKey 2023-04-12 02:55:35,696 INFO [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testInvalidSplitKeyincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 02:55:35,698 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-12 02:55:35,698 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-12 02:55:35,698 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:55:35,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-12 02:55:35,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-12 02:55:35,698 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-12 02:55:35,698 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:55:35,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-12 02:55:35,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 02:55:35,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 02:55:35,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-12 02:55:35,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-12 02:55:35,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:55:35,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:55:35,699 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:55:35,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 02:55:35,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:55:35,699 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:35,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-12 02:55:35,699 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-12 02:55:35,699 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:55:35,699 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-12 02:55:35,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-12 02:55:35,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-12 02:55:35,699 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:55:35,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:55:35,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testInvalidSplitKey 2023-04-12 02:55:35,700 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:35,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 02:55:35,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-12 02:55:35,700 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:35,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testInvalidSplitKey 2023-04-12 02:55:35,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:35,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:35,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:35,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 02:55:35,702 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:35,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testInvalidSplitKey 2023-04-12 02:55:35,702 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:35,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:35,702 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:35,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:35,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:35,704 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:35,705 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-04-12 02:55:35,705 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:55:35,705 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:55:35,705 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:55:35,705 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:55:35,705 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:55:35,705 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:55:35,705 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:55:35,705 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:55:35,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:55:35,705 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:35,705 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:35,705 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-04-12 02:55:35,705 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:35,705 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-12 02:55:35,705 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:35,705 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:35,705 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-12 02:55:35,705 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-12 02:55:35,706 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:55:35,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:55:35,705 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testInvalidSplitKey' 2023-04-12 02:55:35,708 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 02:55:35,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:55:35,705 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:55:35,705 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:55:35,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:55:35,709 DEBUG [Listener at localhost/46667] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testInvalidSplitKey'' to complete. (max 20000 ms per retry) 2023-04-12 02:55:35,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:55:35,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:55:35,709 DEBUG [Listener at localhost/46667] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 02:55:35,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:55:35,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:55:35,809 DEBUG [Listener at localhost/46667] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 02:55:35,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 02:55:35,818 INFO [Listener at localhost/46667] assignment.SplitTableRegionProcedure(206): testInvalidSplitKey,,1681268133998.5add8dc92f700932b95abfa5a81f714d. splitKey isn't explicitly specified, will try to find a best split key from RS jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:35,819 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=34579] regionserver.StoreUtils(129): cannot split hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testInvalidSplitKey/5add8dc92f700932b95abfa5a81f714d/cf1/0417b231dbbf483eb8332c94d8ffdcb4 because midkey is the same as first or last row 2023-04-12 02:55:35,819 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=34579] regionserver.StoreUtils(129): cannot split hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testInvalidSplitKey/5add8dc92f700932b95abfa5a81f714d/cf2/79d3f4e6c7594df9b9c2aed9c5d983b8 because midkey is the same as first or last row 2023-04-12 02:55:35,819 INFO [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=34579] regionserver.HRegion(2745): Flushing 5add8dc92f700932b95abfa5a81f714d 2/2 column families, dataSize=480 B heapSize=2.13 KB 2023-04-12 02:55:35,828 INFO [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=34579] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=240 B at sequenceid=82 (bloomFilter=true), to=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testInvalidSplitKey/5add8dc92f700932b95abfa5a81f714d/.tmp/cf1/35b2648247fe4d65a6c367086869571c 2023-04-12 02:55:35,850 INFO [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=34579] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=240 B at sequenceid=82 (bloomFilter=true), to=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testInvalidSplitKey/5add8dc92f700932b95abfa5a81f714d/.tmp/cf2/e0518934c7414482abd87a6388b5224c 2023-04-12 02:55:35,855 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=34579] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testInvalidSplitKey/5add8dc92f700932b95abfa5a81f714d/.tmp/cf1/35b2648247fe4d65a6c367086869571c as hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testInvalidSplitKey/5add8dc92f700932b95abfa5a81f714d/cf1/35b2648247fe4d65a6c367086869571c 2023-04-12 02:55:35,859 INFO [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=34579] regionserver.HStore(1080): Added hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testInvalidSplitKey/5add8dc92f700932b95abfa5a81f714d/cf1/35b2648247fe4d65a6c367086869571c, entries=8, sequenceid=82, filesize=5.0 K 2023-04-12 02:55:35,860 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=34579] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testInvalidSplitKey/5add8dc92f700932b95abfa5a81f714d/.tmp/cf2/e0518934c7414482abd87a6388b5224c as hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testInvalidSplitKey/5add8dc92f700932b95abfa5a81f714d/cf2/e0518934c7414482abd87a6388b5224c 2023-04-12 02:55:35,864 INFO [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=34579] regionserver.HStore(1080): Added hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testInvalidSplitKey/5add8dc92f700932b95abfa5a81f714d/cf2/e0518934c7414482abd87a6388b5224c, entries=8, sequenceid=82, filesize=5.0 K 2023-04-12 02:55:35,865 INFO [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=34579] regionserver.HRegion(2948): Finished flush of dataSize ~480 B/480, heapSize ~2.09 KB/2144, currentSize=0 B/0 for 5add8dc92f700932b95abfa5a81f714d in 46ms, sequenceid=82, compaction requested=true 2023-04-12 02:55:35,866 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=34579] regionserver.HRegion(2446): Flush status journal for 5add8dc92f700932b95abfa5a81f714d: 2023-04-12 02:55:35,866 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=34579] regionserver.StoreUtils(129): cannot split hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testInvalidSplitKey/5add8dc92f700932b95abfa5a81f714d/cf1/0417b231dbbf483eb8332c94d8ffdcb4 because midkey is the same as first or last row 2023-04-12 02:55:35,866 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=34579] regionserver.StoreUtils(129): cannot split hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testInvalidSplitKey/5add8dc92f700932b95abfa5a81f714d/cf2/79d3f4e6c7594df9b9c2aed9c5d983b8 because midkey is the same as first or last row 2023-04-12 02:55:35,867 DEBUG [Listener at localhost/46667] assignment.SplitTableRegionProcedure(219): Splittable=true state=OPEN, location=jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:35,867 DEBUG [Listener at localhost/46667] assignment.TestSplitTableRegionProcedure(438): Expected Split procedure construction failure: Region not splittable because bestSplitPoint = null, maybe table is too small for auto split. For force split, try specifying split row 2023-04-12 02:55:35,867 WARN [Listener at localhost/46667] procedure2.ProcedureTestingUtility(193): Set Kill before store update to: false 2023-04-12 02:55:35,873 INFO [Listener at localhost/46667] client.HBaseAdmin$15(890): Started disable of testInvalidSplitKey 2023-04-12 02:55:35,874 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable testInvalidSplitKey 2023-04-12 02:55:35,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] procedure2.ProcedureExecutor(1029): Stored pid=105, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testInvalidSplitKey 2023-04-12 02:55:35,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] master.MasterRpcServices(1227): Checking to see if procedure is done pid=105 2023-04-12 02:55:35,879 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testInvalidSplitKey","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681268135878"}]},"ts":"1681268135878"} 2023-04-12 02:55:35,880 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testInvalidSplitKey, state=DISABLING in hbase:meta 2023-04-12 02:55:35,882 INFO [PEWorker-1] procedure.DisableTableProcedure(293): Set testInvalidSplitKey to state=DISABLING 2023-04-12 02:55:35,883 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=106, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testInvalidSplitKey, region=5add8dc92f700932b95abfa5a81f714d, UNASSIGN}] 2023-04-12 02:55:35,884 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=106, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testInvalidSplitKey, region=5add8dc92f700932b95abfa5a81f714d, UNASSIGN 2023-04-12 02:55:35,885 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=106 updating hbase:meta row=5add8dc92f700932b95abfa5a81f714d, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:35,885 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testInvalidSplitKey,,1681268133998.5add8dc92f700932b95abfa5a81f714d.","families":{"info":[{"qualifier":"regioninfo","vlen":53,"tag":[],"timestamp":"1681268135885"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681268135885"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681268135885"}]},"ts":"1681268135885"} 2023-04-12 02:55:35,887 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=107, ppid=106, state=RUNNABLE; CloseRegionProcedure 5add8dc92f700932b95abfa5a81f714d, server=jenkins-hbase4.apache.org,34579,1681268031719}] 2023-04-12 02:55:35,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] master.MasterRpcServices(1227): Checking to see if procedure is done pid=105 2023-04-12 02:55:36,039 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 5add8dc92f700932b95abfa5a81f714d 2023-04-12 02:55:36,039 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 5add8dc92f700932b95abfa5a81f714d, disabling compactions & flushes 2023-04-12 02:55:36,039 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region testInvalidSplitKey,,1681268133998.5add8dc92f700932b95abfa5a81f714d. 2023-04-12 02:55:36,039 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testInvalidSplitKey,,1681268133998.5add8dc92f700932b95abfa5a81f714d. 2023-04-12 02:55:36,039 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on testInvalidSplitKey,,1681268133998.5add8dc92f700932b95abfa5a81f714d. after waiting 0 ms 2023-04-12 02:55:36,039 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region testInvalidSplitKey,,1681268133998.5add8dc92f700932b95abfa5a81f714d. 2023-04-12 02:55:36,044 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testInvalidSplitKey/5add8dc92f700932b95abfa5a81f714d/recovered.edits/85.seqid, newMaxSeqId=85, maxSeqId=1 2023-04-12 02:55:36,045 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro 2023-04-12 02:55:36,045 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed testInvalidSplitKey,,1681268133998.5add8dc92f700932b95abfa5a81f714d. 2023-04-12 02:55:36,045 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 5add8dc92f700932b95abfa5a81f714d: 2023-04-12 02:55:36,047 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 5add8dc92f700932b95abfa5a81f714d 2023-04-12 02:55:36,047 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=106 updating hbase:meta row=5add8dc92f700932b95abfa5a81f714d, regionState=CLOSED 2023-04-12 02:55:36,047 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testInvalidSplitKey,,1681268133998.5add8dc92f700932b95abfa5a81f714d.","families":{"info":[{"qualifier":"regioninfo","vlen":53,"tag":[],"timestamp":"1681268136047"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681268136047"}]},"ts":"1681268136047"} 2023-04-12 02:55:36,050 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=107, resume processing ppid=106 2023-04-12 02:55:36,050 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=107, ppid=106, state=SUCCESS; CloseRegionProcedure 5add8dc92f700932b95abfa5a81f714d, server=jenkins-hbase4.apache.org,34579,1681268031719 in 161 msec 2023-04-12 02:55:36,052 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=106, resume processing ppid=105 2023-04-12 02:55:36,052 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=106, ppid=105, state=SUCCESS; TransitRegionStateProcedure table=testInvalidSplitKey, region=5add8dc92f700932b95abfa5a81f714d, UNASSIGN in 167 msec 2023-04-12 02:55:36,052 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testInvalidSplitKey","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681268136052"}]},"ts":"1681268136052"} 2023-04-12 02:55:36,053 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testInvalidSplitKey, state=DISABLED in hbase:meta 2023-04-12 02:55:36,055 INFO [PEWorker-1] procedure.DisableTableProcedure(305): Set testInvalidSplitKey to state=DISABLED 2023-04-12 02:55:36,056 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=105, state=SUCCESS; DisableTableProcedure table=testInvalidSplitKey in 181 msec 2023-04-12 02:55:36,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] master.MasterRpcServices(1227): Checking to see if procedure is done pid=105 2023-04-12 02:55:36,181 INFO [Listener at localhost/46667] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testInvalidSplitKey, procId: 105 completed 2023-04-12 02:55:36,182 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete testInvalidSplitKey 2023-04-12 02:55:36,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] procedure2.ProcedureExecutor(1029): Stored pid=108, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=testInvalidSplitKey 2023-04-12 02:55:36,184 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=108, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=testInvalidSplitKey 2023-04-12 02:55:36,185 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=108, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=testInvalidSplitKey 2023-04-12 02:55:36,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] master.MasterRpcServices(1227): Checking to see if procedure is done pid=108 2023-04-12 02:55:36,188 DEBUG [HFileArchiver-11] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testInvalidSplitKey/5add8dc92f700932b95abfa5a81f714d 2023-04-12 02:55:36,191 DEBUG [HFileArchiver-11] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testInvalidSplitKey/5add8dc92f700932b95abfa5a81f714d/cf1, FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testInvalidSplitKey/5add8dc92f700932b95abfa5a81f714d/cf2, FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testInvalidSplitKey/5add8dc92f700932b95abfa5a81f714d/recovered.edits] 2023-04-12 02:55:36,195 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testInvalidSplitKey/5add8dc92f700932b95abfa5a81f714d/cf1/0417b231dbbf483eb8332c94d8ffdcb4 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testInvalidSplitKey/5add8dc92f700932b95abfa5a81f714d/cf1/0417b231dbbf483eb8332c94d8ffdcb4 2023-04-12 02:55:36,197 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testInvalidSplitKey/5add8dc92f700932b95abfa5a81f714d/cf1/35b2648247fe4d65a6c367086869571c to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testInvalidSplitKey/5add8dc92f700932b95abfa5a81f714d/cf1/35b2648247fe4d65a6c367086869571c 2023-04-12 02:55:36,198 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testInvalidSplitKey/5add8dc92f700932b95abfa5a81f714d/cf1/53389b263afe426aa1bc1b611cf318b8 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testInvalidSplitKey/5add8dc92f700932b95abfa5a81f714d/cf1/53389b263afe426aa1bc1b611cf318b8 2023-04-12 02:55:36,200 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testInvalidSplitKey/5add8dc92f700932b95abfa5a81f714d/cf1/58c01e644f78443db3caaba4ffc33a47 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testInvalidSplitKey/5add8dc92f700932b95abfa5a81f714d/cf1/58c01e644f78443db3caaba4ffc33a47 2023-04-12 02:55:36,201 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testInvalidSplitKey/5add8dc92f700932b95abfa5a81f714d/cf1/638abb3c0a454a38824724abf4d1490a to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testInvalidSplitKey/5add8dc92f700932b95abfa5a81f714d/cf1/638abb3c0a454a38824724abf4d1490a 2023-04-12 02:55:36,202 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testInvalidSplitKey/5add8dc92f700932b95abfa5a81f714d/cf1/a9e4df0381a9416bab6903def5777a58 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testInvalidSplitKey/5add8dc92f700932b95abfa5a81f714d/cf1/a9e4df0381a9416bab6903def5777a58 2023-04-12 02:55:36,203 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testInvalidSplitKey/5add8dc92f700932b95abfa5a81f714d/cf1/fcd93194ff764d89b7767e7ac2433220 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testInvalidSplitKey/5add8dc92f700932b95abfa5a81f714d/cf1/fcd93194ff764d89b7767e7ac2433220 2023-04-12 02:55:36,206 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testInvalidSplitKey/5add8dc92f700932b95abfa5a81f714d/cf2/79d3f4e6c7594df9b9c2aed9c5d983b8 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testInvalidSplitKey/5add8dc92f700932b95abfa5a81f714d/cf2/79d3f4e6c7594df9b9c2aed9c5d983b8 2023-04-12 02:55:36,208 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testInvalidSplitKey/5add8dc92f700932b95abfa5a81f714d/cf2/7e77ef73cda643b5b2908483142515d6 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testInvalidSplitKey/5add8dc92f700932b95abfa5a81f714d/cf2/7e77ef73cda643b5b2908483142515d6 2023-04-12 02:55:36,209 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testInvalidSplitKey/5add8dc92f700932b95abfa5a81f714d/cf2/9b376f96ee5d42108c970007d7a46c72 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testInvalidSplitKey/5add8dc92f700932b95abfa5a81f714d/cf2/9b376f96ee5d42108c970007d7a46c72 2023-04-12 02:55:36,210 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testInvalidSplitKey/5add8dc92f700932b95abfa5a81f714d/cf2/bc014d4c1b4e4357bfb0a84cd5632dd8 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testInvalidSplitKey/5add8dc92f700932b95abfa5a81f714d/cf2/bc014d4c1b4e4357bfb0a84cd5632dd8 2023-04-12 02:55:36,211 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testInvalidSplitKey/5add8dc92f700932b95abfa5a81f714d/cf2/c882be8d0ad34abf9c16dce48a03abeb to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testInvalidSplitKey/5add8dc92f700932b95abfa5a81f714d/cf2/c882be8d0ad34abf9c16dce48a03abeb 2023-04-12 02:55:36,212 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testInvalidSplitKey/5add8dc92f700932b95abfa5a81f714d/cf2/e0518934c7414482abd87a6388b5224c to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testInvalidSplitKey/5add8dc92f700932b95abfa5a81f714d/cf2/e0518934c7414482abd87a6388b5224c 2023-04-12 02:55:36,214 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testInvalidSplitKey/5add8dc92f700932b95abfa5a81f714d/cf2/e2288421480743338fec14aaced87369 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testInvalidSplitKey/5add8dc92f700932b95abfa5a81f714d/cf2/e2288421480743338fec14aaced87369 2023-04-12 02:55:36,216 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testInvalidSplitKey/5add8dc92f700932b95abfa5a81f714d/recovered.edits/85.seqid to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testInvalidSplitKey/5add8dc92f700932b95abfa5a81f714d/recovered.edits/85.seqid 2023-04-12 02:55:36,217 DEBUG [HFileArchiver-11] backup.HFileArchiver(596): Deleted hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testInvalidSplitKey/5add8dc92f700932b95abfa5a81f714d 2023-04-12 02:55:36,217 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived testInvalidSplitKey regions 2023-04-12 02:55:36,219 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=108, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=testInvalidSplitKey 2023-04-12 02:55:36,220 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of testInvalidSplitKey from hbase:meta 2023-04-12 02:55:36,224 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'testInvalidSplitKey' descriptor. 2023-04-12 02:55:36,225 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=108, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=testInvalidSplitKey 2023-04-12 02:55:36,225 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'testInvalidSplitKey' from region states. 2023-04-12 02:55:36,226 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testInvalidSplitKey,,1681268133998.5add8dc92f700932b95abfa5a81f714d.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1681268136225"}]},"ts":"9223372036854775807"} 2023-04-12 02:55:36,227 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-12 02:55:36,227 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 5add8dc92f700932b95abfa5a81f714d, NAME => 'testInvalidSplitKey,,1681268133998.5add8dc92f700932b95abfa5a81f714d.', STARTKEY => '', ENDKEY => ''}] 2023-04-12 02:55:36,227 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'testInvalidSplitKey' as deleted. 2023-04-12 02:55:36,227 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testInvalidSplitKey","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1681268136227"}]},"ts":"9223372036854775807"} 2023-04-12 02:55:36,229 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table testInvalidSplitKey state from META 2023-04-12 02:55:36,231 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=108, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=testInvalidSplitKey 2023-04-12 02:55:36,232 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=108, state=SUCCESS; DeleteTableProcedure table=testInvalidSplitKey in 49 msec 2023-04-12 02:55:36,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] master.MasterRpcServices(1227): Checking to see if procedure is done pid=108 2023-04-12 02:55:36,287 INFO [Listener at localhost/46667] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:testInvalidSplitKey, procId: 108 completed 2023-04-12 02:55:36,303 INFO [Listener at localhost/46667] hbase.ResourceChecker(175): after: master.assignment.TestSplitTableRegionProcedure#testInvalidSplitKey Thread=365 (was 345) - Thread LEAK? -, OpenFileDescriptor=736 (was 649) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=314 (was 314), ProcessCount=172 (was 172), AvailableMemoryMB=836 (was 843) 2023-04-12 02:55:36,318 INFO [Listener at localhost/46667] hbase.ResourceChecker(147): before: master.assignment.TestSplitTableRegionProcedure#testRecoveryAndDoubleExecution Thread=365, OpenFileDescriptor=736, MaxFileDescriptor=60000, SystemLoadAverage=314, ProcessCount=172, AvailableMemoryMB=836 2023-04-12 02:55:36,318 WARN [Listener at localhost/46667] procedure2.ProcedureTestingUtility(193): Set Kill before store update to: false 2023-04-12 02:55:36,320 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/balancer 2023-04-12 02:55:36,320 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] master.MasterRpcServices(492): Client=jenkins//172.31.14.131 set balanceSwitch=false 2023-04-12 02:55:36,321 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(1029): Stored pid=109, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=testRecoveryAndDoubleExecution 2023-04-12 02:55:36,323 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=109, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=testRecoveryAndDoubleExecution execute state=CREATE_TABLE_PRE_OPERATION 2023-04-12 02:55:36,324 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=109, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=testRecoveryAndDoubleExecution execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-12 02:55:36,325 DEBUG [HFileArchiver-12] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61 2023-04-12 02:55:36,326 DEBUG [HFileArchiver-12] backup.HFileArchiver(153): Directory hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61 empty. 2023-04-12 02:55:36,326 DEBUG [HFileArchiver-12] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61 2023-04-12 02:55:36,326 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived testRecoveryAndDoubleExecution regions 2023-04-12 02:55:36,337 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testRecoveryAndDoubleExecution/.tabledesc/.tableinfo.0000000001 2023-04-12 02:55:36,338 INFO [RegionOpenAndInit-testRecoveryAndDoubleExecution-pool-0] regionserver.HRegion(7675): creating {ENCODED => c09b5f7c44d4a83602090cbccc082b61, NAME => 'testRecoveryAndDoubleExecution,,1681268136320.c09b5f7c44d4a83602090cbccc082b61.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='testRecoveryAndDoubleExecution', {NAME => 'cf1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'cf2', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp 2023-04-12 02:55:36,348 DEBUG [RegionOpenAndInit-testRecoveryAndDoubleExecution-pool-0] regionserver.HRegion(866): Instantiated testRecoveryAndDoubleExecution,,1681268136320.c09b5f7c44d4a83602090cbccc082b61.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-12 02:55:36,349 DEBUG [RegionOpenAndInit-testRecoveryAndDoubleExecution-pool-0] regionserver.HRegion(1604): Closing c09b5f7c44d4a83602090cbccc082b61, disabling compactions & flushes 2023-04-12 02:55:36,349 INFO [RegionOpenAndInit-testRecoveryAndDoubleExecution-pool-0] regionserver.HRegion(1626): Closing region testRecoveryAndDoubleExecution,,1681268136320.c09b5f7c44d4a83602090cbccc082b61. 2023-04-12 02:55:36,349 DEBUG [RegionOpenAndInit-testRecoveryAndDoubleExecution-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testRecoveryAndDoubleExecution,,1681268136320.c09b5f7c44d4a83602090cbccc082b61. 2023-04-12 02:55:36,349 DEBUG [RegionOpenAndInit-testRecoveryAndDoubleExecution-pool-0] regionserver.HRegion(1714): Acquired close lock on testRecoveryAndDoubleExecution,,1681268136320.c09b5f7c44d4a83602090cbccc082b61. after waiting 0 ms 2023-04-12 02:55:36,349 DEBUG [RegionOpenAndInit-testRecoveryAndDoubleExecution-pool-0] regionserver.HRegion(1724): Updates disabled for region testRecoveryAndDoubleExecution,,1681268136320.c09b5f7c44d4a83602090cbccc082b61. 2023-04-12 02:55:36,349 INFO [RegionOpenAndInit-testRecoveryAndDoubleExecution-pool-0] regionserver.HRegion(1838): Closed testRecoveryAndDoubleExecution,,1681268136320.c09b5f7c44d4a83602090cbccc082b61. 2023-04-12 02:55:36,349 DEBUG [RegionOpenAndInit-testRecoveryAndDoubleExecution-pool-0] regionserver.HRegion(1558): Region close journal for c09b5f7c44d4a83602090cbccc082b61: 2023-04-12 02:55:36,352 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=109, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=testRecoveryAndDoubleExecution execute state=CREATE_TABLE_ADD_TO_META 2023-04-12 02:55:36,353 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRecoveryAndDoubleExecution,,1681268136320.c09b5f7c44d4a83602090cbccc082b61.","families":{"info":[{"qualifier":"regioninfo","vlen":64,"tag":[],"timestamp":"1681268136353"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681268136353"}]},"ts":"1681268136353"} 2023-04-12 02:55:36,354 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-12 02:55:36,355 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=109, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=testRecoveryAndDoubleExecution execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-12 02:55:36,355 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRecoveryAndDoubleExecution","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681268136355"}]},"ts":"1681268136355"} 2023-04-12 02:55:36,356 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testRecoveryAndDoubleExecution, state=ENABLING in hbase:meta 2023-04-12 02:55:36,361 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-04-12 02:55:36,364 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-04-12 02:55:36,364 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-04-12 02:55:36,364 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-04-12 02:55:36,364 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-04-12 02:55:36,364 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=110, ppid=109, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=c09b5f7c44d4a83602090cbccc082b61, ASSIGN}] 2023-04-12 02:55:36,367 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=110, ppid=109, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=c09b5f7c44d4a83602090cbccc082b61, ASSIGN 2023-04-12 02:55:36,368 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=110, ppid=109, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=c09b5f7c44d4a83602090cbccc082b61, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,43091,1681268031804; forceNewPlan=false, retain=false 2023-04-12 02:55:36,518 INFO [jenkins-hbase4:40281] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-04-12 02:55:36,519 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=110 updating hbase:meta row=c09b5f7c44d4a83602090cbccc082b61, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:36,519 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRecoveryAndDoubleExecution,,1681268136320.c09b5f7c44d4a83602090cbccc082b61.","families":{"info":[{"qualifier":"regioninfo","vlen":64,"tag":[],"timestamp":"1681268136519"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681268136519"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681268136519"}]},"ts":"1681268136519"} 2023-04-12 02:55:36,521 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=111, ppid=110, state=RUNNABLE; OpenRegionProcedure c09b5f7c44d4a83602090cbccc082b61, server=jenkins-hbase4.apache.org,43091,1681268031804}] 2023-04-12 02:55:36,678 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRecoveryAndDoubleExecution,,1681268136320.c09b5f7c44d4a83602090cbccc082b61. 2023-04-12 02:55:36,678 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => c09b5f7c44d4a83602090cbccc082b61, NAME => 'testRecoveryAndDoubleExecution,,1681268136320.c09b5f7c44d4a83602090cbccc082b61.', STARTKEY => '', ENDKEY => ''} 2023-04-12 02:55:36,678 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(174): System coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro loaded, priority=536870911. 2023-04-12 02:55:36,678 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRecoveryAndDoubleExecution c09b5f7c44d4a83602090cbccc082b61 2023-04-12 02:55:36,679 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated testRecoveryAndDoubleExecution,,1681268136320.c09b5f7c44d4a83602090cbccc082b61.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-12 02:55:36,679 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for c09b5f7c44d4a83602090cbccc082b61 2023-04-12 02:55:36,679 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for c09b5f7c44d4a83602090cbccc082b61 2023-04-12 02:55:36,680 INFO [StoreOpener-c09b5f7c44d4a83602090cbccc082b61-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf1 of region c09b5f7c44d4a83602090cbccc082b61 2023-04-12 02:55:36,681 DEBUG [StoreOpener-c09b5f7c44d4a83602090cbccc082b61-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf1 2023-04-12 02:55:36,681 DEBUG [StoreOpener-c09b5f7c44d4a83602090cbccc082b61-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf1 2023-04-12 02:55:36,682 INFO [StoreOpener-c09b5f7c44d4a83602090cbccc082b61-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region c09b5f7c44d4a83602090cbccc082b61 columnFamilyName cf1 2023-04-12 02:55:36,682 INFO [StoreOpener-c09b5f7c44d4a83602090cbccc082b61-1] regionserver.HStore(310): Store=c09b5f7c44d4a83602090cbccc082b61/cf1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-12 02:55:36,683 INFO [StoreOpener-c09b5f7c44d4a83602090cbccc082b61-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf2 of region c09b5f7c44d4a83602090cbccc082b61 2023-04-12 02:55:36,684 DEBUG [StoreOpener-c09b5f7c44d4a83602090cbccc082b61-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf2 2023-04-12 02:55:36,684 DEBUG [StoreOpener-c09b5f7c44d4a83602090cbccc082b61-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf2 2023-04-12 02:55:36,684 INFO [StoreOpener-c09b5f7c44d4a83602090cbccc082b61-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region c09b5f7c44d4a83602090cbccc082b61 columnFamilyName cf2 2023-04-12 02:55:36,685 INFO [StoreOpener-c09b5f7c44d4a83602090cbccc082b61-1] regionserver.HStore(310): Store=c09b5f7c44d4a83602090cbccc082b61/cf2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-12 02:55:36,686 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61 2023-04-12 02:55:36,686 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61 2023-04-12 02:55:36,688 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testRecoveryAndDoubleExecution descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-04-12 02:55:36,689 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for c09b5f7c44d4a83602090cbccc082b61 2023-04-12 02:55:36,690 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-12 02:55:36,691 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened c09b5f7c44d4a83602090cbccc082b61; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=63827343, jitterRate=-0.048898473381996155}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-04-12 02:55:36,691 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for c09b5f7c44d4a83602090cbccc082b61: 2023-04-12 02:55:36,692 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRecoveryAndDoubleExecution,,1681268136320.c09b5f7c44d4a83602090cbccc082b61., pid=111, masterSystemTime=1681268136675 2023-04-12 02:55:36,694 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRecoveryAndDoubleExecution,,1681268136320.c09b5f7c44d4a83602090cbccc082b61. 2023-04-12 02:55:36,694 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRecoveryAndDoubleExecution,,1681268136320.c09b5f7c44d4a83602090cbccc082b61. 2023-04-12 02:55:36,694 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=110 updating hbase:meta row=c09b5f7c44d4a83602090cbccc082b61, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:36,694 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRecoveryAndDoubleExecution,,1681268136320.c09b5f7c44d4a83602090cbccc082b61.","families":{"info":[{"qualifier":"regioninfo","vlen":64,"tag":[],"timestamp":"1681268136694"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1681268136694"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681268136694"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681268136694"}]},"ts":"1681268136694"} 2023-04-12 02:55:36,697 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=111, resume processing ppid=110 2023-04-12 02:55:36,697 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=111, ppid=110, state=SUCCESS; OpenRegionProcedure c09b5f7c44d4a83602090cbccc082b61, server=jenkins-hbase4.apache.org,43091,1681268031804 in 174 msec 2023-04-12 02:55:36,699 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=110, resume processing ppid=109 2023-04-12 02:55:36,699 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=110, ppid=109, state=SUCCESS; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=c09b5f7c44d4a83602090cbccc082b61, ASSIGN in 333 msec 2023-04-12 02:55:36,700 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=109, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=testRecoveryAndDoubleExecution execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-12 02:55:36,700 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRecoveryAndDoubleExecution","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681268136700"}]},"ts":"1681268136700"} 2023-04-12 02:55:36,703 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testRecoveryAndDoubleExecution, state=ENABLED in hbase:meta 2023-04-12 02:55:36,714 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=109, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=testRecoveryAndDoubleExecution execute state=CREATE_TABLE_POST_OPERATION 2023-04-12 02:55:36,715 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=109, state=SUCCESS; CreateTableProcedure table=testRecoveryAndDoubleExecution in 393 msec 2023-04-12 02:55:36,824 DEBUG [Listener at localhost/46667] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-12 02:55:36,828 INFO [RS-EventLoopGroup-5-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:38048, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-12 02:55:36,836 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 02:55:36,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] procedure.ProcedureCoordinator(165): Submitting procedure testRecoveryAndDoubleExecution 2023-04-12 02:55:36,838 INFO [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testRecoveryAndDoubleExecution' 2023-04-12 02:55:36,838 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:55:36,838 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testRecoveryAndDoubleExecution' starting 'acquire' 2023-04-12 02:55:36,838 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testRecoveryAndDoubleExecution', kicking off acquire phase on members. 2023-04-12 02:55:36,838 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-12 02:55:36,838 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-04-12 02:55:36,842 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:55:36,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:55:36,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:55:36,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-04-12 02:55:36,844 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-12 02:55:36,844 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:55:36,844 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:55:36,844 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:55:36,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:55:36,844 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:55:36,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:55:36,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 02:55:36,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-04-12 02:55:36,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-04-12 02:55:36,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-04-12 02:55:36,845 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testRecoveryAndDoubleExecution 2023-04-12 02:55:36,845 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testRecoveryAndDoubleExecution 2023-04-12 02:55:36,845 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-12 02:55:36,845 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testRecoveryAndDoubleExecution' with timeout 60000ms 2023-04-12 02:55:36,845 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:55:36,845 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-12 02:55:36,845 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testRecoveryAndDoubleExecution' starting 'acquire' stage 2023-04-12 02:55:36,845 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testRecoveryAndDoubleExecution' locally acquired 2023-04-12 02:55:36,845 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34579,1681268031719' joining acquired barrier for procedure (testRecoveryAndDoubleExecution) in zk 2023-04-12 02:55:36,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 02:55:36,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-04-12 02:55:36,846 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testRecoveryAndDoubleExecution 2023-04-12 02:55:36,846 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testRecoveryAndDoubleExecution 2023-04-12 02:55:36,846 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testRecoveryAndDoubleExecution' with timeout 60000ms 2023-04-12 02:55:36,846 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:55:36,846 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testRecoveryAndDoubleExecution' starting 'acquire' stage 2023-04-12 02:55:36,846 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testRecoveryAndDoubleExecution' locally acquired 2023-04-12 02:55:36,846 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44679,1681268031772' joining acquired barrier for procedure (testRecoveryAndDoubleExecution) in zk 2023-04-12 02:55:36,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 02:55:36,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-04-12 02:55:36,847 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testRecoveryAndDoubleExecution 2023-04-12 02:55:36,848 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:36,849 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-12 02:55:36,850 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-12 02:55:36,850 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-12 02:55:36,850 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testRecoveryAndDoubleExecution' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 02:55:36,850 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:36,850 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 02:55:36,850 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-12 02:55:36,850 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testRecoveryAndDoubleExecution' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 02:55:36,859 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testRecoveryAndDoubleExecution 2023-04-12 02:55:36,859 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testRecoveryAndDoubleExecution' with timeout 60000ms 2023-04-12 02:55:36,859 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:55:36,859 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testRecoveryAndDoubleExecution' starting 'acquire' stage 2023-04-12 02:55:36,860 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 02:55:36,860 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 02:55:36,860 DEBUG [rs(jenkins-hbase4.apache.org,43091,1681268031804)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testRecoveryAndDoubleExecution,,1681268136320.c09b5f7c44d4a83602090cbccc082b61. 2023-04-12 02:55:36,860 DEBUG [rs(jenkins-hbase4.apache.org,43091,1681268031804)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testRecoveryAndDoubleExecution,,1681268136320.c09b5f7c44d4a83602090cbccc082b61. started... 2023-04-12 02:55:36,862 INFO [rs(jenkins-hbase4.apache.org,43091,1681268031804)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c09b5f7c44d4a83602090cbccc082b61 2/2 column families, dataSize=120 B heapSize=928 B 2023-04-12 02:55:36,882 INFO [rs(jenkins-hbase4.apache.org,43091,1681268031804)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=60 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/.tmp/cf1/a79c7cc5c59c4f6bbe6a6a584d824f36 2023-04-12 02:55:36,907 INFO [rs(jenkins-hbase4.apache.org,43091,1681268031804)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=60 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/.tmp/cf2/5b22fe6156a04c8c9486fd6f189bf4d7 2023-04-12 02:55:36,912 DEBUG [rs(jenkins-hbase4.apache.org,43091,1681268031804)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/.tmp/cf1/a79c7cc5c59c4f6bbe6a6a584d824f36 as hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf1/a79c7cc5c59c4f6bbe6a6a584d824f36 2023-04-12 02:55:36,916 INFO [rs(jenkins-hbase4.apache.org,43091,1681268031804)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf1/a79c7cc5c59c4f6bbe6a6a584d824f36, entries=2, sequenceid=6, filesize=4.8 K 2023-04-12 02:55:36,917 DEBUG [rs(jenkins-hbase4.apache.org,43091,1681268031804)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/.tmp/cf2/5b22fe6156a04c8c9486fd6f189bf4d7 as hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf2/5b22fe6156a04c8c9486fd6f189bf4d7 2023-04-12 02:55:36,921 INFO [rs(jenkins-hbase4.apache.org,43091,1681268031804)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf2/5b22fe6156a04c8c9486fd6f189bf4d7, entries=2, sequenceid=6, filesize=4.8 K 2023-04-12 02:55:36,922 INFO [rs(jenkins-hbase4.apache.org,43091,1681268031804)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~120 B/120, heapSize ~896 B/896, currentSize=0 B/0 for c09b5f7c44d4a83602090cbccc082b61 in 59ms, sequenceid=6, compaction requested=false 2023-04-12 02:55:36,922 DEBUG [rs(jenkins-hbase4.apache.org,43091,1681268031804)-flush-proc-pool-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'testRecoveryAndDoubleExecution' 2023-04-12 02:55:36,922 DEBUG [rs(jenkins-hbase4.apache.org,43091,1681268031804)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c09b5f7c44d4a83602090cbccc082b61: 2023-04-12 02:55:36,922 DEBUG [rs(jenkins-hbase4.apache.org,43091,1681268031804)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testRecoveryAndDoubleExecution,,1681268136320.c09b5f7c44d4a83602090cbccc082b61. 2023-04-12 02:55:36,922 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 02:55:36,922 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 02:55:36,923 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:36,923 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testRecoveryAndDoubleExecution' locally acquired 2023-04-12 02:55:36,923 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43091,1681268031804' joining acquired barrier for procedure (testRecoveryAndDoubleExecution) in zk 2023-04-12 02:55:36,924 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:36,924 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-12 02:55:36,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:36,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 02:55:36,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 02:55:36,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 02:55:36,925 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-12 02:55:36,925 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testRecoveryAndDoubleExecution' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 02:55:36,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 02:55:36,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRecoveryAndDoubleExecution 2023-04-12 02:55:36,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:36,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:36,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:36,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 02:55:36,927 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43091,1681268031804' joining acquired barrier for procedure 'testRecoveryAndDoubleExecution' on coordinator 2023-04-12 02:55:36,927 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@37d81459[Count = 0] remaining members to acquire global barrier 2023-04-12 02:55:36,927 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testRecoveryAndDoubleExecution' starting 'in-barrier' execution. 2023-04-12 02:55:36,927 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-12 02:55:36,928 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-12 02:55:36,928 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-12 02:55:36,928 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-12 02:55:36,928 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-12 02:55:36,928 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-12 02:55:36,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-12 02:55:36,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-12 02:55:36,928 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testRecoveryAndDoubleExecution' received 'reached' from coordinator. 2023-04-12 02:55:36,928 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testRecoveryAndDoubleExecution' received 'reached' from coordinator. 2023-04-12 02:55:36,928 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-12 02:55:36,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-12 02:55:36,928 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testRecoveryAndDoubleExecution' locally completed 2023-04-12 02:55:36,928 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testRecoveryAndDoubleExecution' locally completed 2023-04-12 02:55:36,929 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testRecoveryAndDoubleExecution' completed for member 'jenkins-hbase4.apache.org,44679,1681268031772' in zk 2023-04-12 02:55:36,928 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:36,929 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 02:55:36,929 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testRecoveryAndDoubleExecution' completed for member 'jenkins-hbase4.apache.org,43091,1681268031804' in zk 2023-04-12 02:55:36,929 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testRecoveryAndDoubleExecution' received 'reached' from coordinator. 2023-04-12 02:55:36,929 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testRecoveryAndDoubleExecution' locally completed 2023-04-12 02:55:36,929 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testRecoveryAndDoubleExecution' completed for member 'jenkins-hbase4.apache.org,34579,1681268031719' in zk 2023-04-12 02:55:36,932 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testRecoveryAndDoubleExecution' has notified controller of completion 2023-04-12 02:55:36,932 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:55:36,932 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testRecoveryAndDoubleExecution' has notified controller of completion 2023-04-12 02:55:36,932 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:55:36,932 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testRecoveryAndDoubleExecution' completed. 2023-04-12 02:55:36,932 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testRecoveryAndDoubleExecution' completed. 2023-04-12 02:55:36,932 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:36,932 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testRecoveryAndDoubleExecution' has notified controller of completion 2023-04-12 02:55:36,932 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:55:36,932 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testRecoveryAndDoubleExecution' completed. 2023-04-12 02:55:36,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:36,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 02:55:36,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 02:55:36,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 02:55:36,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 02:55:36,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRecoveryAndDoubleExecution 2023-04-12 02:55:36,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:36,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:36,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:36,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 02:55:36,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRecoveryAndDoubleExecution 2023-04-12 02:55:36,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:36,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:36,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:36,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testRecoveryAndDoubleExecution' member 'jenkins-hbase4.apache.org,43091,1681268031804': 2023-04-12 02:55:36,937 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43091,1681268031804' released barrier for procedure'testRecoveryAndDoubleExecution', counting down latch. Waiting for 0 more 2023-04-12 02:55:36,937 INFO [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testRecoveryAndDoubleExecution' execution completed 2023-04-12 02:55:36,937 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 02:55:36,937 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 02:55:36,937 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testRecoveryAndDoubleExecution 2023-04-12 02:55:36,937 INFO [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testRecoveryAndDoubleExecutionincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 02:55:36,939 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-12 02:55:36,939 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-12 02:55:36,939 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-12 02:55:36,939 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-12 02:55:36,939 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-12 02:55:36,939 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:55:36,939 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:55:36,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-12 02:55:36,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 02:55:36,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 02:55:36,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-12 02:55:36,939 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-12 02:55:36,939 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:55:36,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-12 02:55:36,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 02:55:36,939 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:36,940 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:55:36,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:55:36,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRecoveryAndDoubleExecution 2023-04-12 02:55:36,940 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:55:36,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:55:36,940 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:36,940 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-12 02:55:36,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-12 02:55:36,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-12 02:55:36,941 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:36,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-12 02:55:36,941 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:55:36,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:55:36,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 02:55:36,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-12 02:55:36,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRecoveryAndDoubleExecution 2023-04-12 02:55:36,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:36,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:36,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:36,943 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:36,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 02:55:36,943 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:36,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRecoveryAndDoubleExecution 2023-04-12 02:55:36,943 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:36,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:36,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:36,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:36,946 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:55:36,946 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:36,946 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:55:36,946 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-04-12 02:55:36,946 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:55:36,946 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:55:36,946 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:55:36,946 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:55:36,947 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testRecoveryAndDoubleExecution' 2023-04-12 02:55:36,947 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 02:55:36,946 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:55:36,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:55:36,946 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:36,947 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:36,947 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:55:36,947 DEBUG [Listener at localhost/46667] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testRecoveryAndDoubleExecution'' to complete. (max 20000 ms per retry) 2023-04-12 02:55:36,946 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:55:36,947 DEBUG [Listener at localhost/46667] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 02:55:36,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:55:36,947 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-04-12 02:55:36,947 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:55:36,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:55:36,947 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:36,947 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:55:36,947 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-12 02:55:36,947 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:36,947 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:36,947 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-12 02:55:36,947 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-12 02:55:36,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:55:36,947 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:55:36,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:55:36,948 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:55:36,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:55:37,047 DEBUG [Listener at localhost/46667] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 02:55:37,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 02:55:37,062 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 02:55:37,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] procedure.ProcedureCoordinator(143): Procedure testRecoveryAndDoubleExecution was in running list but was completed. Accepting new attempt. 2023-04-12 02:55:37,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] procedure.ProcedureCoordinator(165): Submitting procedure testRecoveryAndDoubleExecution 2023-04-12 02:55:37,064 INFO [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testRecoveryAndDoubleExecution' 2023-04-12 02:55:37,064 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:55:37,064 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testRecoveryAndDoubleExecution' starting 'acquire' 2023-04-12 02:55:37,064 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testRecoveryAndDoubleExecution', kicking off acquire phase on members. 2023-04-12 02:55:37,064 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,064 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,067 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:55:37,067 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:37,067 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:55:37,067 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:55:37,067 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:55:37,067 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:55:37,067 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:55:37,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:55:37,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:55:37,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:55:37,068 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:37,068 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 02:55:37,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,068 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,069 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 02:55:37,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,069 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testRecoveryAndDoubleExecution 2023-04-12 02:55:37,069 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testRecoveryAndDoubleExecution is still present, removing 2023-04-12 02:55:37,069 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testRecoveryAndDoubleExecution 2023-04-12 02:55:37,069 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testRecoveryAndDoubleExecution' with timeout 60000ms 2023-04-12 02:55:37,069 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:55:37,069 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testRecoveryAndDoubleExecution' starting 'acquire' stage 2023-04-12 02:55:37,069 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testRecoveryAndDoubleExecution' locally acquired 2023-04-12 02:55:37,069 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44679,1681268031772' joining acquired barrier for procedure (testRecoveryAndDoubleExecution) in zk 2023-04-12 02:55:37,070 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 02:55:37,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,070 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testRecoveryAndDoubleExecution 2023-04-12 02:55:37,070 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testRecoveryAndDoubleExecution is still present, removing 2023-04-12 02:55:37,070 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testRecoveryAndDoubleExecution 2023-04-12 02:55:37,070 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testRecoveryAndDoubleExecution' with timeout 60000ms 2023-04-12 02:55:37,071 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:55:37,071 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testRecoveryAndDoubleExecution' starting 'acquire' stage 2023-04-12 02:55:37,071 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 02:55:37,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,071 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testRecoveryAndDoubleExecution 2023-04-12 02:55:37,071 DEBUG [rs(jenkins-hbase4.apache.org,43091,1681268031804)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testRecoveryAndDoubleExecution,,1681268136320.c09b5f7c44d4a83602090cbccc082b61. 2023-04-12 02:55:37,071 DEBUG [rs(jenkins-hbase4.apache.org,43091,1681268031804)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testRecoveryAndDoubleExecution,,1681268136320.c09b5f7c44d4a83602090cbccc082b61. started... 2023-04-12 02:55:37,071 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 02:55:37,071 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 02:55:37,071 INFO [rs(jenkins-hbase4.apache.org,43091,1681268031804)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c09b5f7c44d4a83602090cbccc082b61 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-04-12 02:55:37,071 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,071 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testRecoveryAndDoubleExecution' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 02:55:37,076 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testRecoveryAndDoubleExecution is still present, removing 2023-04-12 02:55:37,076 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testRecoveryAndDoubleExecution 2023-04-12 02:55:37,076 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testRecoveryAndDoubleExecution' with timeout 60000ms 2023-04-12 02:55:37,076 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:55:37,076 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testRecoveryAndDoubleExecution' starting 'acquire' stage 2023-04-12 02:55:37,076 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testRecoveryAndDoubleExecution' locally acquired 2023-04-12 02:55:37,076 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34579,1681268031719' joining acquired barrier for procedure (testRecoveryAndDoubleExecution) in zk 2023-04-12 02:55:37,078 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,078 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,078 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testRecoveryAndDoubleExecution' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 02:55:37,089 INFO [rs(jenkins-hbase4.apache.org,43091,1681268031804)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=19 (bloomFilter=true), to=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/.tmp/cf1/63025bd542924f2caced3f22c416a5a1 2023-04-12 02:55:37,102 INFO [rs(jenkins-hbase4.apache.org,43091,1681268031804)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=19 (bloomFilter=true), to=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/.tmp/cf2/a4e6a82328b24916838ccb970ddd7704 2023-04-12 02:55:37,106 DEBUG [rs(jenkins-hbase4.apache.org,43091,1681268031804)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/.tmp/cf1/63025bd542924f2caced3f22c416a5a1 as hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf1/63025bd542924f2caced3f22c416a5a1 2023-04-12 02:55:37,110 INFO [rs(jenkins-hbase4.apache.org,43091,1681268031804)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf1/63025bd542924f2caced3f22c416a5a1, entries=10, sequenceid=19, filesize=5.0 K 2023-04-12 02:55:37,111 DEBUG [rs(jenkins-hbase4.apache.org,43091,1681268031804)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/.tmp/cf2/a4e6a82328b24916838ccb970ddd7704 as hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf2/a4e6a82328b24916838ccb970ddd7704 2023-04-12 02:55:37,116 INFO [rs(jenkins-hbase4.apache.org,43091,1681268031804)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf2/a4e6a82328b24916838ccb970ddd7704, entries=10, sequenceid=19, filesize=5.0 K 2023-04-12 02:55:37,117 INFO [rs(jenkins-hbase4.apache.org,43091,1681268031804)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for c09b5f7c44d4a83602090cbccc082b61 in 46ms, sequenceid=19, compaction requested=false 2023-04-12 02:55:37,117 DEBUG [rs(jenkins-hbase4.apache.org,43091,1681268031804)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c09b5f7c44d4a83602090cbccc082b61: 2023-04-12 02:55:37,117 DEBUG [rs(jenkins-hbase4.apache.org,43091,1681268031804)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testRecoveryAndDoubleExecution,,1681268136320.c09b5f7c44d4a83602090cbccc082b61. 2023-04-12 02:55:37,117 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 02:55:37,117 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 02:55:37,117 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:37,117 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testRecoveryAndDoubleExecution' locally acquired 2023-04-12 02:55:37,117 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43091,1681268031804' joining acquired barrier for procedure (testRecoveryAndDoubleExecution) in zk 2023-04-12 02:55:37,124 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:37,124 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:37,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 02:55:37,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 02:55:37,124 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,124 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testRecoveryAndDoubleExecution' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 02:55:37,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 02:55:37,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 02:55:37,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRecoveryAndDoubleExecution 2023-04-12 02:55:37,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:37,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:37,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:37,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 02:55:37,126 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43091,1681268031804' joining acquired barrier for procedure 'testRecoveryAndDoubleExecution' on coordinator 2023-04-12 02:55:37,127 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6fdef063[Count = 0] remaining members to acquire global barrier 2023-04-12 02:55:37,127 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testRecoveryAndDoubleExecution' starting 'in-barrier' execution. 2023-04-12 02:55:37,127 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,128 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,128 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,128 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,128 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,128 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,128 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,128 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testRecoveryAndDoubleExecution' received 'reached' from coordinator. 2023-04-12 02:55:37,128 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testRecoveryAndDoubleExecution' locally completed 2023-04-12 02:55:37,128 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testRecoveryAndDoubleExecution' completed for member 'jenkins-hbase4.apache.org,34579,1681268031719' in zk 2023-04-12 02:55:37,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,128 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testRecoveryAndDoubleExecution' received 'reached' from coordinator. 2023-04-12 02:55:37,129 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testRecoveryAndDoubleExecution' locally completed 2023-04-12 02:55:37,129 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testRecoveryAndDoubleExecution' received 'reached' from coordinator. 2023-04-12 02:55:37,129 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testRecoveryAndDoubleExecution' locally completed 2023-04-12 02:55:37,129 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testRecoveryAndDoubleExecution' completed for member 'jenkins-hbase4.apache.org,43091,1681268031804' in zk 2023-04-12 02:55:37,129 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:37,129 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 02:55:37,129 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testRecoveryAndDoubleExecution' completed for member 'jenkins-hbase4.apache.org,44679,1681268031772' in zk 2023-04-12 02:55:37,130 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testRecoveryAndDoubleExecution' has notified controller of completion 2023-04-12 02:55:37,130 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:55:37,130 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testRecoveryAndDoubleExecution' completed. 2023-04-12 02:55:37,131 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testRecoveryAndDoubleExecution' has notified controller of completion 2023-04-12 02:55:37,131 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:37,131 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testRecoveryAndDoubleExecution' has notified controller of completion 2023-04-12 02:55:37,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:37,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 02:55:37,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 02:55:37,131 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:55:37,131 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:55:37,131 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testRecoveryAndDoubleExecution' completed. 2023-04-12 02:55:37,131 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testRecoveryAndDoubleExecution' completed. 2023-04-12 02:55:37,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 02:55:37,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 02:55:37,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRecoveryAndDoubleExecution 2023-04-12 02:55:37,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:37,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:37,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:37,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 02:55:37,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRecoveryAndDoubleExecution 2023-04-12 02:55:37,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:37,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:37,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:37,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testRecoveryAndDoubleExecution' member 'jenkins-hbase4.apache.org,43091,1681268031804': 2023-04-12 02:55:37,136 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43091,1681268031804' released barrier for procedure'testRecoveryAndDoubleExecution', counting down latch. Waiting for 0 more 2023-04-12 02:55:37,136 INFO [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testRecoveryAndDoubleExecution' execution completed 2023-04-12 02:55:37,136 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 02:55:37,136 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 02:55:37,136 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testRecoveryAndDoubleExecution 2023-04-12 02:55:37,136 INFO [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testRecoveryAndDoubleExecutionincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 02:55:37,139 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,139 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:55:37,139 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,139 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,139 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,139 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 02:55:37,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 02:55:37,139 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,139 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:55:37,139 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,139 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:55:37,139 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:37,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,139 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:55:37,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 02:55:37,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:55:37,140 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:37,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,141 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:55:37,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:55:37,141 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:37,140 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:55:37,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRecoveryAndDoubleExecution 2023-04-12 02:55:37,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:55:37,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 02:55:37,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRecoveryAndDoubleExecution 2023-04-12 02:55:37,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:37,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:37,143 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:37,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:37,143 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:37,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 02:55:37,143 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:37,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRecoveryAndDoubleExecution 2023-04-12 02:55:37,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:37,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:37,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:37,146 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:55:37,146 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:55:37,146 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:55:37,146 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:55:37,146 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:37,146 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:55:37,146 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:55:37,146 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:55:37,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:55:37,146 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:55:37,146 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:55:37,146 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testRecoveryAndDoubleExecution' 2023-04-12 02:55:37,147 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:55:37,146 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:37,147 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:55:37,147 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 02:55:37,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:55:37,147 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:37,147 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,147 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:37,147 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,147 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:37,147 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:37,147 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,147 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,147 DEBUG [Listener at localhost/46667] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testRecoveryAndDoubleExecution'' to complete. (max 20000 ms per retry) 2023-04-12 02:55:37,147 DEBUG [Listener at localhost/46667] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 02:55:37,147 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:55:37,147 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:55:37,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:55:37,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:55:37,148 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:55:37,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:55:37,247 DEBUG [Listener at localhost/46667] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 02:55:37,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 02:55:37,250 ERROR [BP-1247239137-172.31.14.131-1681268026808 heartbeating to localhost/127.0.0.1:44691] datanode.BPServiceActor(848): Exception in BPOfferService for Block pool BP-1247239137-172.31.14.131-1681268026808 (Datanode Uuid 79922b64-644d-4757-a58c-9707ee400c82) service to localhost/127.0.0.1:44691 java.lang.IllegalThreadStateException at java.lang.ThreadGroup.addUnstarted(ThreadGroup.java:867) at java.lang.Thread.init(Thread.java:407) at java.lang.Thread.init(Thread.java:351) at java.lang.Thread.(Thread.java:501) at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService$1.newThread(FsDatasetAsyncDiskService.java:102) at java.util.concurrent.ThreadPoolExecutor$Worker.(ThreadPoolExecutor.java:619) at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:932) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1367) at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService.execute(FsDatasetAsyncDiskService.java:172) at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService.deleteAsync(FsDatasetAsyncDiskService.java:221) at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.invalidate(FsDatasetImpl.java:2216) at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.invalidate(FsDatasetImpl.java:2140) at org.apache.hadoop.hdfs.server.datanode.BPOfferService.processCommandFromActive(BPOfferService.java:732) at org.apache.hadoop.hdfs.server.datanode.BPOfferService.processCommandFromActor(BPOfferService.java:679) at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.processCommand(BPServiceActor.java:880) at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:676) at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:846) at java.lang.Thread.run(Thread.java:750) 2023-04-12 02:55:37,264 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 02:55:37,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] procedure.ProcedureCoordinator(143): Procedure testRecoveryAndDoubleExecution was in running list but was completed. Accepting new attempt. 2023-04-12 02:55:37,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] procedure.ProcedureCoordinator(165): Submitting procedure testRecoveryAndDoubleExecution 2023-04-12 02:55:37,266 INFO [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testRecoveryAndDoubleExecution' 2023-04-12 02:55:37,266 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:55:37,266 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testRecoveryAndDoubleExecution' starting 'acquire' 2023-04-12 02:55:37,266 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testRecoveryAndDoubleExecution', kicking off acquire phase on members. 2023-04-12 02:55:37,266 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,266 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,267 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:55:37,267 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:55:37,267 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:55:37,267 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:37,267 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:55:37,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:55:37,267 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:55:37,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:55:37,268 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:55:37,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:55:37,268 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:37,268 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 02:55:37,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,269 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,269 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,269 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 02:55:37,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,269 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testRecoveryAndDoubleExecution 2023-04-12 02:55:37,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 02:55:37,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,269 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testRecoveryAndDoubleExecution 2023-04-12 02:55:37,269 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testRecoveryAndDoubleExecution is still present, removing 2023-04-12 02:55:37,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 02:55:37,270 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testRecoveryAndDoubleExecution is still present, removing 2023-04-12 02:55:37,270 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testRecoveryAndDoubleExecution 2023-04-12 02:55:37,270 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testRecoveryAndDoubleExecution 2023-04-12 02:55:37,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,270 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testRecoveryAndDoubleExecution' with timeout 60000ms 2023-04-12 02:55:37,270 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testRecoveryAndDoubleExecution 2023-04-12 02:55:37,270 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:55:37,270 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testRecoveryAndDoubleExecution' starting 'acquire' stage 2023-04-12 02:55:37,270 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testRecoveryAndDoubleExecution' locally acquired 2023-04-12 02:55:37,270 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44679,1681268031772' joining acquired barrier for procedure (testRecoveryAndDoubleExecution) in zk 2023-04-12 02:55:37,270 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testRecoveryAndDoubleExecution' with timeout 60000ms 2023-04-12 02:55:37,270 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:55:37,270 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testRecoveryAndDoubleExecution' starting 'acquire' stage 2023-04-12 02:55:37,276 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testRecoveryAndDoubleExecution is still present, removing 2023-04-12 02:55:37,276 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testRecoveryAndDoubleExecution 2023-04-12 02:55:37,276 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testRecoveryAndDoubleExecution' with timeout 60000ms 2023-04-12 02:55:37,276 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:55:37,276 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testRecoveryAndDoubleExecution' starting 'acquire' stage 2023-04-12 02:55:37,276 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testRecoveryAndDoubleExecution' locally acquired 2023-04-12 02:55:37,276 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34579,1681268031719' joining acquired barrier for procedure (testRecoveryAndDoubleExecution) in zk 2023-04-12 02:55:37,278 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,280 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 02:55:37,280 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 02:55:37,280 DEBUG [rs(jenkins-hbase4.apache.org,43091,1681268031804)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testRecoveryAndDoubleExecution,,1681268136320.c09b5f7c44d4a83602090cbccc082b61. 2023-04-12 02:55:37,280 DEBUG [rs(jenkins-hbase4.apache.org,43091,1681268031804)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testRecoveryAndDoubleExecution,,1681268136320.c09b5f7c44d4a83602090cbccc082b61. started... 2023-04-12 02:55:37,280 INFO [rs(jenkins-hbase4.apache.org,43091,1681268031804)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c09b5f7c44d4a83602090cbccc082b61 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-04-12 02:55:37,280 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,280 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,281 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testRecoveryAndDoubleExecution' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 02:55:37,281 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,281 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testRecoveryAndDoubleExecution' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 02:55:37,300 INFO [rs(jenkins-hbase4.apache.org,43091,1681268031804)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=32 (bloomFilter=true), to=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/.tmp/cf1/9d5ec6ededfd4de8b6d44e635d18346c 2023-04-12 02:55:37,320 INFO [rs(jenkins-hbase4.apache.org,43091,1681268031804)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=32 (bloomFilter=true), to=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/.tmp/cf2/fb8ce8dc74bd47fabd120f6f688962d5 2023-04-12 02:55:37,329 DEBUG [rs(jenkins-hbase4.apache.org,43091,1681268031804)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/.tmp/cf1/9d5ec6ededfd4de8b6d44e635d18346c as hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf1/9d5ec6ededfd4de8b6d44e635d18346c 2023-04-12 02:55:37,333 INFO [rs(jenkins-hbase4.apache.org,43091,1681268031804)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf1/9d5ec6ededfd4de8b6d44e635d18346c, entries=10, sequenceid=32, filesize=5.0 K 2023-04-12 02:55:37,333 DEBUG [rs(jenkins-hbase4.apache.org,43091,1681268031804)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/.tmp/cf2/fb8ce8dc74bd47fabd120f6f688962d5 as hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf2/fb8ce8dc74bd47fabd120f6f688962d5 2023-04-12 02:55:37,336 INFO [rs(jenkins-hbase4.apache.org,43091,1681268031804)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf2/fb8ce8dc74bd47fabd120f6f688962d5, entries=10, sequenceid=32, filesize=5.0 K 2023-04-12 02:55:37,337 INFO [rs(jenkins-hbase4.apache.org,43091,1681268031804)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for c09b5f7c44d4a83602090cbccc082b61 in 57ms, sequenceid=32, compaction requested=true 2023-04-12 02:55:37,337 DEBUG [rs(jenkins-hbase4.apache.org,43091,1681268031804)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c09b5f7c44d4a83602090cbccc082b61: 2023-04-12 02:55:37,337 DEBUG [rs(jenkins-hbase4.apache.org,43091,1681268031804)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testRecoveryAndDoubleExecution,,1681268136320.c09b5f7c44d4a83602090cbccc082b61. 2023-04-12 02:55:37,337 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 02:55:37,338 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 02:55:37,338 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:37,338 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testRecoveryAndDoubleExecution' locally acquired 2023-04-12 02:55:37,338 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43091,1681268031804' joining acquired barrier for procedure (testRecoveryAndDoubleExecution) in zk 2023-04-12 02:55:37,339 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:37,339 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:37,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 02:55:37,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 02:55:37,340 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,340 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testRecoveryAndDoubleExecution' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 02:55:37,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 02:55:37,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 02:55:37,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRecoveryAndDoubleExecution 2023-04-12 02:55:37,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:37,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:37,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:37,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 02:55:37,342 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43091,1681268031804' joining acquired barrier for procedure 'testRecoveryAndDoubleExecution' on coordinator 2023-04-12 02:55:37,342 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5982cc31[Count = 0] remaining members to acquire global barrier 2023-04-12 02:55:37,342 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testRecoveryAndDoubleExecution' starting 'in-barrier' execution. 2023-04-12 02:55:37,342 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,343 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,343 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,343 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,343 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,343 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,343 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:37,343 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testRecoveryAndDoubleExecution' received 'reached' from coordinator. 2023-04-12 02:55:37,343 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testRecoveryAndDoubleExecution' locally completed 2023-04-12 02:55:37,343 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testRecoveryAndDoubleExecution' completed for member 'jenkins-hbase4.apache.org,43091,1681268031804' in zk 2023-04-12 02:55:37,343 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 02:55:37,343 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testRecoveryAndDoubleExecution' received 'reached' from coordinator. 2023-04-12 02:55:37,343 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,344 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testRecoveryAndDoubleExecution' locally completed 2023-04-12 02:55:37,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,344 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testRecoveryAndDoubleExecution' completed for member 'jenkins-hbase4.apache.org,34579,1681268031719' in zk 2023-04-12 02:55:37,344 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testRecoveryAndDoubleExecution' received 'reached' from coordinator. 2023-04-12 02:55:37,344 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testRecoveryAndDoubleExecution' locally completed 2023-04-12 02:55:37,344 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testRecoveryAndDoubleExecution' completed for member 'jenkins-hbase4.apache.org,44679,1681268031772' in zk 2023-04-12 02:55:37,345 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:37,345 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testRecoveryAndDoubleExecution' has notified controller of completion 2023-04-12 02:55:37,345 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:55:37,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:37,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 02:55:37,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 02:55:37,345 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testRecoveryAndDoubleExecution' completed. 2023-04-12 02:55:37,347 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testRecoveryAndDoubleExecution' has notified controller of completion 2023-04-12 02:55:37,347 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:55:37,347 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testRecoveryAndDoubleExecution' completed. 2023-04-12 02:55:37,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 02:55:37,347 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testRecoveryAndDoubleExecution' has notified controller of completion 2023-04-12 02:55:37,348 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:55:37,348 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testRecoveryAndDoubleExecution' completed. 2023-04-12 02:55:37,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 02:55:37,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRecoveryAndDoubleExecution 2023-04-12 02:55:37,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:37,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:37,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:37,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 02:55:37,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRecoveryAndDoubleExecution 2023-04-12 02:55:37,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:37,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:37,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:37,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testRecoveryAndDoubleExecution' member 'jenkins-hbase4.apache.org,43091,1681268031804': 2023-04-12 02:55:37,351 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43091,1681268031804' released barrier for procedure'testRecoveryAndDoubleExecution', counting down latch. Waiting for 0 more 2023-04-12 02:55:37,351 INFO [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testRecoveryAndDoubleExecution' execution completed 2023-04-12 02:55:37,351 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 02:55:37,351 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 02:55:37,351 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testRecoveryAndDoubleExecution 2023-04-12 02:55:37,351 INFO [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testRecoveryAndDoubleExecutionincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 02:55:37,352 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,352 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,353 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,353 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,353 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,353 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:55:37,353 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,353 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:55:37,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 02:55:37,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 02:55:37,353 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:55:37,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,353 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,353 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:37,353 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:55:37,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:55:37,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 02:55:37,353 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:55:37,354 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:37,353 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:55:37,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:55:37,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:55:37,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRecoveryAndDoubleExecution 2023-04-12 02:55:37,354 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:37,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 02:55:37,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRecoveryAndDoubleExecution 2023-04-12 02:55:37,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:37,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:37,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:37,356 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:37,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 02:55:37,356 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:37,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRecoveryAndDoubleExecution 2023-04-12 02:55:37,356 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:37,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:37,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:37,363 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:37,363 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:55:37,363 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,363 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:55:37,363 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:55:37,363 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:55:37,363 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:55:37,363 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:55:37,364 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:55:37,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:37,363 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:37,363 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:55:37,364 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:37,364 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,364 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:37,364 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testRecoveryAndDoubleExecution' 2023-04-12 02:55:37,364 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 02:55:37,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:55:37,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:55:37,363 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:55:37,363 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:55:37,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:55:37,364 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,364 DEBUG [Listener at localhost/46667] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testRecoveryAndDoubleExecution'' to complete. (max 20000 ms per retry) 2023-04-12 02:55:37,364 DEBUG [Listener at localhost/46667] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 02:55:37,364 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:37,365 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:37,365 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,365 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,368 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:55:37,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:55:37,368 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:55:37,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:55:37,369 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:55:37,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:55:37,369 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution because node does not exist (not an error) 2023-04-12 02:55:37,465 DEBUG [Listener at localhost/46667] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 02:55:37,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 02:55:37,477 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 02:55:37,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] procedure.ProcedureCoordinator(143): Procedure testRecoveryAndDoubleExecution was in running list but was completed. Accepting new attempt. 2023-04-12 02:55:37,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] procedure.ProcedureCoordinator(165): Submitting procedure testRecoveryAndDoubleExecution 2023-04-12 02:55:37,479 INFO [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testRecoveryAndDoubleExecution' 2023-04-12 02:55:37,479 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:55:37,479 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testRecoveryAndDoubleExecution' starting 'acquire' 2023-04-12 02:55:37,479 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testRecoveryAndDoubleExecution', kicking off acquire phase on members. 2023-04-12 02:55:37,480 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,480 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,481 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:55:37,481 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:55:37,481 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:55:37,481 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:55:37,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:55:37,481 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:55:37,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:55:37,481 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:55:37,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:55:37,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,482 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,482 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,482 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:37,482 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 02:55:37,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 02:55:37,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,483 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testRecoveryAndDoubleExecution 2023-04-12 02:55:37,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 02:55:37,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,483 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testRecoveryAndDoubleExecution 2023-04-12 02:55:37,483 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testRecoveryAndDoubleExecution 2023-04-12 02:55:37,483 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:37,483 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 02:55:37,483 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testRecoveryAndDoubleExecution is still present, removing 2023-04-12 02:55:37,483 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testRecoveryAndDoubleExecution 2023-04-12 02:55:37,483 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testRecoveryAndDoubleExecution is still present, removing 2023-04-12 02:55:37,483 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testRecoveryAndDoubleExecution 2023-04-12 02:55:37,483 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testRecoveryAndDoubleExecution' with timeout 60000ms 2023-04-12 02:55:37,483 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testRecoveryAndDoubleExecution' with timeout 60000ms 2023-04-12 02:55:37,483 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:55:37,483 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:55:37,483 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testRecoveryAndDoubleExecution' starting 'acquire' stage 2023-04-12 02:55:37,483 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testRecoveryAndDoubleExecution' starting 'acquire' stage 2023-04-12 02:55:37,483 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testRecoveryAndDoubleExecution' locally acquired 2023-04-12 02:55:37,484 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44679,1681268031772' joining acquired barrier for procedure (testRecoveryAndDoubleExecution) in zk 2023-04-12 02:55:37,484 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testRecoveryAndDoubleExecution is still present, removing 2023-04-12 02:55:37,484 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testRecoveryAndDoubleExecution 2023-04-12 02:55:37,484 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testRecoveryAndDoubleExecution' with timeout 60000ms 2023-04-12 02:55:37,484 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:55:37,484 DEBUG [rs(jenkins-hbase4.apache.org,43091,1681268031804)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testRecoveryAndDoubleExecution,,1681268136320.c09b5f7c44d4a83602090cbccc082b61. 2023-04-12 02:55:37,484 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 02:55:37,485 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 02:55:37,485 DEBUG [rs(jenkins-hbase4.apache.org,43091,1681268031804)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testRecoveryAndDoubleExecution,,1681268136320.c09b5f7c44d4a83602090cbccc082b61. started... 2023-04-12 02:55:37,484 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testRecoveryAndDoubleExecution' starting 'acquire' stage 2023-04-12 02:55:37,485 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testRecoveryAndDoubleExecution' locally acquired 2023-04-12 02:55:37,485 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34579,1681268031719' joining acquired barrier for procedure (testRecoveryAndDoubleExecution) in zk 2023-04-12 02:55:37,485 INFO [rs(jenkins-hbase4.apache.org,43091,1681268031804)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c09b5f7c44d4a83602090cbccc082b61 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-04-12 02:55:37,486 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,487 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,487 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,487 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testRecoveryAndDoubleExecution' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 02:55:37,487 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,487 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testRecoveryAndDoubleExecution' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 02:55:37,494 INFO [rs(jenkins-hbase4.apache.org,43091,1681268031804)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=45 (bloomFilter=true), to=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/.tmp/cf1/001e134c60444ac2b388f9df6bf5670b 2023-04-12 02:55:37,506 INFO [rs(jenkins-hbase4.apache.org,43091,1681268031804)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=45 (bloomFilter=true), to=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/.tmp/cf2/fe343f29642942069fb8e38396f28358 2023-04-12 02:55:37,510 DEBUG [rs(jenkins-hbase4.apache.org,43091,1681268031804)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/.tmp/cf1/001e134c60444ac2b388f9df6bf5670b as hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf1/001e134c60444ac2b388f9df6bf5670b 2023-04-12 02:55:37,514 INFO [rs(jenkins-hbase4.apache.org,43091,1681268031804)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf1/001e134c60444ac2b388f9df6bf5670b, entries=10, sequenceid=45, filesize=5.0 K 2023-04-12 02:55:37,515 DEBUG [rs(jenkins-hbase4.apache.org,43091,1681268031804)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/.tmp/cf2/fe343f29642942069fb8e38396f28358 as hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf2/fe343f29642942069fb8e38396f28358 2023-04-12 02:55:37,519 INFO [rs(jenkins-hbase4.apache.org,43091,1681268031804)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf2/fe343f29642942069fb8e38396f28358, entries=10, sequenceid=45, filesize=5.0 K 2023-04-12 02:55:37,521 INFO [rs(jenkins-hbase4.apache.org,43091,1681268031804)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for c09b5f7c44d4a83602090cbccc082b61 in 36ms, sequenceid=45, compaction requested=true 2023-04-12 02:55:37,521 DEBUG [rs(jenkins-hbase4.apache.org,43091,1681268031804)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c09b5f7c44d4a83602090cbccc082b61: 2023-04-12 02:55:37,521 DEBUG [rs(jenkins-hbase4.apache.org,43091,1681268031804)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testRecoveryAndDoubleExecution,,1681268136320.c09b5f7c44d4a83602090cbccc082b61. 2023-04-12 02:55:37,521 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 02:55:37,521 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 02:55:37,521 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:37,521 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testRecoveryAndDoubleExecution' locally acquired 2023-04-12 02:55:37,521 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43091,1681268031804' joining acquired barrier for procedure (testRecoveryAndDoubleExecution) in zk 2023-04-12 02:55:37,523 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:37,523 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:37,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 02:55:37,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 02:55:37,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 02:55:37,525 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,525 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testRecoveryAndDoubleExecution' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 02:55:37,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 02:55:37,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRecoveryAndDoubleExecution 2023-04-12 02:55:37,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:37,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:37,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:37,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 02:55:37,527 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43091,1681268031804' joining acquired barrier for procedure 'testRecoveryAndDoubleExecution' on coordinator 2023-04-12 02:55:37,527 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@24c257eb[Count = 0] remaining members to acquire global barrier 2023-04-12 02:55:37,527 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testRecoveryAndDoubleExecution' starting 'in-barrier' execution. 2023-04-12 02:55:37,527 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,528 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,528 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,528 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,528 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,529 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,528 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,529 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testRecoveryAndDoubleExecution' received 'reached' from coordinator. 2023-04-12 02:55:37,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,529 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:37,529 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 02:55:37,529 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testRecoveryAndDoubleExecution' locally completed 2023-04-12 02:55:37,529 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testRecoveryAndDoubleExecution' received 'reached' from coordinator. 2023-04-12 02:55:37,529 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testRecoveryAndDoubleExecution' completed for member 'jenkins-hbase4.apache.org,43091,1681268031804' in zk 2023-04-12 02:55:37,529 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testRecoveryAndDoubleExecution' received 'reached' from coordinator. 2023-04-12 02:55:37,529 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testRecoveryAndDoubleExecution' locally completed 2023-04-12 02:55:37,529 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testRecoveryAndDoubleExecution' completed for member 'jenkins-hbase4.apache.org,34579,1681268031719' in zk 2023-04-12 02:55:37,529 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testRecoveryAndDoubleExecution' locally completed 2023-04-12 02:55:37,529 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testRecoveryAndDoubleExecution' completed for member 'jenkins-hbase4.apache.org,44679,1681268031772' in zk 2023-04-12 02:55:37,531 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testRecoveryAndDoubleExecution' has notified controller of completion 2023-04-12 02:55:37,531 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:55:37,531 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testRecoveryAndDoubleExecution' completed. 2023-04-12 02:55:37,531 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:37,531 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testRecoveryAndDoubleExecution' has notified controller of completion 2023-04-12 02:55:37,532 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:55:37,532 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testRecoveryAndDoubleExecution' completed. 2023-04-12 02:55:37,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:37,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 02:55:37,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 02:55:37,532 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testRecoveryAndDoubleExecution' has notified controller of completion 2023-04-12 02:55:37,532 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:55:37,532 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testRecoveryAndDoubleExecution' completed. 2023-04-12 02:55:37,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 02:55:37,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 02:55:37,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRecoveryAndDoubleExecution 2023-04-12 02:55:37,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:37,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:37,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:37,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 02:55:37,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRecoveryAndDoubleExecution 2023-04-12 02:55:37,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:37,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:37,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:37,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testRecoveryAndDoubleExecution' member 'jenkins-hbase4.apache.org,43091,1681268031804': 2023-04-12 02:55:37,540 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43091,1681268031804' released barrier for procedure'testRecoveryAndDoubleExecution', counting down latch. Waiting for 0 more 2023-04-12 02:55:37,540 INFO [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testRecoveryAndDoubleExecution' execution completed 2023-04-12 02:55:37,540 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 02:55:37,540 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 02:55:37,540 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testRecoveryAndDoubleExecution 2023-04-12 02:55:37,540 INFO [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testRecoveryAndDoubleExecutionincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 02:55:37,546 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,546 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,546 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,546 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:55:37,546 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,546 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,546 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,546 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:55:37,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,546 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,546 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:55:37,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,547 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:55:37,547 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:37,547 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:55:37,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:55:37,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:55:37,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 02:55:37,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 02:55:37,547 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:55:37,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:55:37,547 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:37,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 02:55:37,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,548 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:37,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRecoveryAndDoubleExecution 2023-04-12 02:55:37,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 02:55:37,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRecoveryAndDoubleExecution 2023-04-12 02:55:37,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:37,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:37,549 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:37,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:37,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 02:55:37,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRecoveryAndDoubleExecution 2023-04-12 02:55:37,551 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:37,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:37,551 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:37,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:37,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:37,554 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:55:37,554 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:55:37,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:55:37,554 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:55:37,554 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:55:37,554 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:55:37,554 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:37,554 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,554 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:37,554 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:37,554 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,554 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:37,554 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,554 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:37,554 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:37,554 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,554 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,554 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:55:37,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:55:37,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:55:37,554 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:55:37,554 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:55:37,554 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:55:37,555 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:55:37,555 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testRecoveryAndDoubleExecution' 2023-04-12 02:55:37,555 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 02:55:37,555 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:55:37,555 DEBUG [Listener at localhost/46667] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testRecoveryAndDoubleExecution'' to complete. (max 20000 ms per retry) 2023-04-12 02:55:37,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:55:37,555 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:55:37,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:55:37,556 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:55:37,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:55:37,556 DEBUG [Listener at localhost/46667] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 02:55:37,656 DEBUG [Listener at localhost/46667] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 02:55:37,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 02:55:37,672 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 02:55:37,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] procedure.ProcedureCoordinator(143): Procedure testRecoveryAndDoubleExecution was in running list but was completed. Accepting new attempt. 2023-04-12 02:55:37,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] procedure.ProcedureCoordinator(165): Submitting procedure testRecoveryAndDoubleExecution 2023-04-12 02:55:37,675 INFO [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testRecoveryAndDoubleExecution' 2023-04-12 02:55:37,675 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:55:37,676 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testRecoveryAndDoubleExecution' starting 'acquire' 2023-04-12 02:55:37,676 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testRecoveryAndDoubleExecution', kicking off acquire phase on members. 2023-04-12 02:55:37,676 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,676 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,678 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:55:37,678 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:55:37,678 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:37,678 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:55:37,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:55:37,678 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:55:37,678 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:55:37,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:55:37,678 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:55:37,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:55:37,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,679 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,679 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,680 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 02:55:37,680 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:37,680 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 02:55:37,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 02:55:37,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,680 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testRecoveryAndDoubleExecution 2023-04-12 02:55:37,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 02:55:37,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,680 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testRecoveryAndDoubleExecution 2023-04-12 02:55:37,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,681 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testRecoveryAndDoubleExecution 2023-04-12 02:55:37,684 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testRecoveryAndDoubleExecution is still present, removing 2023-04-12 02:55:37,684 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testRecoveryAndDoubleExecution 2023-04-12 02:55:37,684 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testRecoveryAndDoubleExecution' with timeout 60000ms 2023-04-12 02:55:37,684 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testRecoveryAndDoubleExecution is still present, removing 2023-04-12 02:55:37,684 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testRecoveryAndDoubleExecution is still present, removing 2023-04-12 02:55:37,684 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testRecoveryAndDoubleExecution 2023-04-12 02:55:37,684 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testRecoveryAndDoubleExecution' with timeout 60000ms 2023-04-12 02:55:37,684 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:55:37,684 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testRecoveryAndDoubleExecution' starting 'acquire' stage 2023-04-12 02:55:37,684 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testRecoveryAndDoubleExecution' locally acquired 2023-04-12 02:55:37,684 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44679,1681268031772' joining acquired barrier for procedure (testRecoveryAndDoubleExecution) in zk 2023-04-12 02:55:37,684 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:55:37,684 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testRecoveryAndDoubleExecution 2023-04-12 02:55:37,684 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testRecoveryAndDoubleExecution' starting 'acquire' stage 2023-04-12 02:55:37,685 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testRecoveryAndDoubleExecution' with timeout 60000ms 2023-04-12 02:55:37,685 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:55:37,685 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testRecoveryAndDoubleExecution' starting 'acquire' stage 2023-04-12 02:55:37,685 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testRecoveryAndDoubleExecution' locally acquired 2023-04-12 02:55:37,685 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34579,1681268031719' joining acquired barrier for procedure (testRecoveryAndDoubleExecution) in zk 2023-04-12 02:55:37,685 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 02:55:37,685 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 02:55:37,685 DEBUG [rs(jenkins-hbase4.apache.org,43091,1681268031804)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testRecoveryAndDoubleExecution,,1681268136320.c09b5f7c44d4a83602090cbccc082b61. 2023-04-12 02:55:37,685 DEBUG [rs(jenkins-hbase4.apache.org,43091,1681268031804)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testRecoveryAndDoubleExecution,,1681268136320.c09b5f7c44d4a83602090cbccc082b61. started... 2023-04-12 02:55:37,686 INFO [rs(jenkins-hbase4.apache.org,43091,1681268031804)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c09b5f7c44d4a83602090cbccc082b61 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-04-12 02:55:37,687 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,687 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,687 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testRecoveryAndDoubleExecution' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 02:55:37,689 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,690 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,690 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testRecoveryAndDoubleExecution' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 02:55:37,718 INFO [rs(jenkins-hbase4.apache.org,43091,1681268031804)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=58 (bloomFilter=true), to=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/.tmp/cf1/c14bd2fe66004d6d842c72f4b2a4ee90 2023-04-12 02:55:37,732 INFO [rs(jenkins-hbase4.apache.org,43091,1681268031804)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=58 (bloomFilter=true), to=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/.tmp/cf2/1b3fc837d85449cba8b914aa7f18f0fe 2023-04-12 02:55:37,743 DEBUG [rs(jenkins-hbase4.apache.org,43091,1681268031804)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/.tmp/cf1/c14bd2fe66004d6d842c72f4b2a4ee90 as hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf1/c14bd2fe66004d6d842c72f4b2a4ee90 2023-04-12 02:55:37,747 INFO [rs(jenkins-hbase4.apache.org,43091,1681268031804)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf1/c14bd2fe66004d6d842c72f4b2a4ee90, entries=10, sequenceid=58, filesize=5.0 K 2023-04-12 02:55:37,748 DEBUG [rs(jenkins-hbase4.apache.org,43091,1681268031804)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/.tmp/cf2/1b3fc837d85449cba8b914aa7f18f0fe as hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf2/1b3fc837d85449cba8b914aa7f18f0fe 2023-04-12 02:55:37,751 INFO [rs(jenkins-hbase4.apache.org,43091,1681268031804)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf2/1b3fc837d85449cba8b914aa7f18f0fe, entries=10, sequenceid=58, filesize=5.0 K 2023-04-12 02:55:37,752 INFO [rs(jenkins-hbase4.apache.org,43091,1681268031804)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for c09b5f7c44d4a83602090cbccc082b61 in 67ms, sequenceid=58, compaction requested=true 2023-04-12 02:55:37,752 DEBUG [rs(jenkins-hbase4.apache.org,43091,1681268031804)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c09b5f7c44d4a83602090cbccc082b61: 2023-04-12 02:55:37,752 DEBUG [rs(jenkins-hbase4.apache.org,43091,1681268031804)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testRecoveryAndDoubleExecution,,1681268136320.c09b5f7c44d4a83602090cbccc082b61. 2023-04-12 02:55:37,752 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 02:55:37,753 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 02:55:37,753 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:37,753 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testRecoveryAndDoubleExecution' locally acquired 2023-04-12 02:55:37,753 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43091,1681268031804' joining acquired barrier for procedure (testRecoveryAndDoubleExecution) in zk 2023-04-12 02:55:37,757 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:37,758 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:37,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 02:55:37,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 02:55:37,758 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,758 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testRecoveryAndDoubleExecution' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 02:55:37,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 02:55:37,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 02:55:37,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRecoveryAndDoubleExecution 2023-04-12 02:55:37,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:37,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:37,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:37,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 02:55:37,760 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43091,1681268031804' joining acquired barrier for procedure 'testRecoveryAndDoubleExecution' on coordinator 2023-04-12 02:55:37,760 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@53cff5f6[Count = 0] remaining members to acquire global barrier 2023-04-12 02:55:37,760 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testRecoveryAndDoubleExecution' starting 'in-barrier' execution. 2023-04-12 02:55:37,760 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,761 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,761 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,761 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,761 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testRecoveryAndDoubleExecution' received 'reached' from coordinator. 2023-04-12 02:55:37,761 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testRecoveryAndDoubleExecution' locally completed 2023-04-12 02:55:37,761 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testRecoveryAndDoubleExecution' completed for member 'jenkins-hbase4.apache.org,44679,1681268031772' in zk 2023-04-12 02:55:37,761 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,761 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,762 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testRecoveryAndDoubleExecution' received 'reached' from coordinator. 2023-04-12 02:55:37,762 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testRecoveryAndDoubleExecution' locally completed 2023-04-12 02:55:37,762 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testRecoveryAndDoubleExecution' completed for member 'jenkins-hbase4.apache.org,43091,1681268031804' in zk 2023-04-12 02:55:37,762 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,762 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testRecoveryAndDoubleExecution' received 'reached' from coordinator. 2023-04-12 02:55:37,762 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testRecoveryAndDoubleExecution' locally completed 2023-04-12 02:55:37,762 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testRecoveryAndDoubleExecution' completed for member 'jenkins-hbase4.apache.org,34579,1681268031719' in zk 2023-04-12 02:55:37,762 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:37,762 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 02:55:37,764 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:37,764 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testRecoveryAndDoubleExecution' has notified controller of completion 2023-04-12 02:55:37,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:37,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 02:55:37,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 02:55:37,764 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testRecoveryAndDoubleExecution' has notified controller of completion 2023-04-12 02:55:37,764 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:55:37,764 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:55:37,765 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testRecoveryAndDoubleExecution' completed. 2023-04-12 02:55:37,765 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testRecoveryAndDoubleExecution' completed. 2023-04-12 02:55:37,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 02:55:37,765 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testRecoveryAndDoubleExecution' has notified controller of completion 2023-04-12 02:55:37,765 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:55:37,765 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testRecoveryAndDoubleExecution' completed. 2023-04-12 02:55:37,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 02:55:37,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRecoveryAndDoubleExecution 2023-04-12 02:55:37,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:37,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:37,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:37,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 02:55:37,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRecoveryAndDoubleExecution 2023-04-12 02:55:37,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:37,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:37,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:37,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testRecoveryAndDoubleExecution' member 'jenkins-hbase4.apache.org,43091,1681268031804': 2023-04-12 02:55:37,769 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43091,1681268031804' released barrier for procedure'testRecoveryAndDoubleExecution', counting down latch. Waiting for 0 more 2023-04-12 02:55:37,769 INFO [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testRecoveryAndDoubleExecution' execution completed 2023-04-12 02:55:37,769 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 02:55:37,769 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 02:55:37,769 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testRecoveryAndDoubleExecution 2023-04-12 02:55:37,769 INFO [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testRecoveryAndDoubleExecutionincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 02:55:37,770 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,770 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,770 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,770 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:55:37,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,770 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,770 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:55:37,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 02:55:37,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 02:55:37,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,770 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:55:37,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,771 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:55:37,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:55:37,771 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:37,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 02:55:37,771 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:55:37,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:55:37,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,771 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:55:37,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:55:37,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,772 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:37,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRecoveryAndDoubleExecution 2023-04-12 02:55:37,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,773 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:37,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 02:55:37,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRecoveryAndDoubleExecution 2023-04-12 02:55:37,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:37,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:37,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:37,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 02:55:37,774 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:37,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRecoveryAndDoubleExecution 2023-04-12 02:55:37,775 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:37,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:37,775 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:37,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:37,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:37,777 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:55:37,778 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:55:37,778 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:55:37,778 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:37,778 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:55:37,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:55:37,778 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:55:37,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:55:37,778 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:55:37,778 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:55:37,778 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testRecoveryAndDoubleExecution' 2023-04-12 02:55:37,778 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 02:55:37,778 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,778 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:55:37,778 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:37,778 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:55:37,778 DEBUG [Listener at localhost/46667] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testRecoveryAndDoubleExecution'' to complete. (max 20000 ms per retry) 2023-04-12 02:55:37,778 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:55:37,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:55:37,779 DEBUG [Listener at localhost/46667] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 02:55:37,778 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:55:37,778 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:37,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:55:37,779 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:55:37,779 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:55:37,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:55:37,779 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:55:37,779 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:37,779 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,779 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:37,779 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:37,779 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,779 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,879 DEBUG [Listener at localhost/46667] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 02:55:37,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 02:55:37,893 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 02:55:37,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] procedure.ProcedureCoordinator(143): Procedure testRecoveryAndDoubleExecution was in running list but was completed. Accepting new attempt. 2023-04-12 02:55:37,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] procedure.ProcedureCoordinator(165): Submitting procedure testRecoveryAndDoubleExecution 2023-04-12 02:55:37,895 INFO [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testRecoveryAndDoubleExecution' 2023-04-12 02:55:37,895 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:55:37,895 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testRecoveryAndDoubleExecution' starting 'acquire' 2023-04-12 02:55:37,895 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testRecoveryAndDoubleExecution', kicking off acquire phase on members. 2023-04-12 02:55:37,896 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,896 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,898 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:55:37,898 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:55:37,898 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:55:37,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:55:37,898 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:37,898 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:55:37,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:55:37,898 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:55:37,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,899 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:55:37,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:55:37,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,899 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:37,899 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 02:55:37,899 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,899 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 02:55:37,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,900 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,900 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testRecoveryAndDoubleExecution 2023-04-12 02:55:37,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 02:55:37,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 02:55:37,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,900 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testRecoveryAndDoubleExecution 2023-04-12 02:55:37,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,900 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testRecoveryAndDoubleExecution is still present, removing 2023-04-12 02:55:37,900 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testRecoveryAndDoubleExecution 2023-04-12 02:55:37,901 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testRecoveryAndDoubleExecution is still present, removing 2023-04-12 02:55:37,901 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testRecoveryAndDoubleExecution 2023-04-12 02:55:37,900 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testRecoveryAndDoubleExecution 2023-04-12 02:55:37,901 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testRecoveryAndDoubleExecution' with timeout 60000ms 2023-04-12 02:55:37,901 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testRecoveryAndDoubleExecution' with timeout 60000ms 2023-04-12 02:55:37,901 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:55:37,901 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testRecoveryAndDoubleExecution' starting 'acquire' stage 2023-04-12 02:55:37,901 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testRecoveryAndDoubleExecution' locally acquired 2023-04-12 02:55:37,901 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44679,1681268031772' joining acquired barrier for procedure (testRecoveryAndDoubleExecution) in zk 2023-04-12 02:55:37,901 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testRecoveryAndDoubleExecution is still present, removing 2023-04-12 02:55:37,901 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testRecoveryAndDoubleExecution 2023-04-12 02:55:37,901 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:55:37,901 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testRecoveryAndDoubleExecution' starting 'acquire' stage 2023-04-12 02:55:37,901 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testRecoveryAndDoubleExecution' with timeout 60000ms 2023-04-12 02:55:37,901 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:55:37,901 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testRecoveryAndDoubleExecution' locally acquired 2023-04-12 02:55:37,901 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34579,1681268031719' joining acquired barrier for procedure (testRecoveryAndDoubleExecution) in zk 2023-04-12 02:55:37,901 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testRecoveryAndDoubleExecution' starting 'acquire' stage 2023-04-12 02:55:37,902 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 02:55:37,902 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 02:55:37,902 DEBUG [rs(jenkins-hbase4.apache.org,43091,1681268031804)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testRecoveryAndDoubleExecution,,1681268136320.c09b5f7c44d4a83602090cbccc082b61. 2023-04-12 02:55:37,902 DEBUG [rs(jenkins-hbase4.apache.org,43091,1681268031804)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testRecoveryAndDoubleExecution,,1681268136320.c09b5f7c44d4a83602090cbccc082b61. started... 2023-04-12 02:55:37,902 INFO [rs(jenkins-hbase4.apache.org,43091,1681268031804)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c09b5f7c44d4a83602090cbccc082b61 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-04-12 02:55:37,903 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,904 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,904 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,904 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testRecoveryAndDoubleExecution' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 02:55:37,905 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,905 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testRecoveryAndDoubleExecution' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 02:55:37,927 INFO [rs(jenkins-hbase4.apache.org,43091,1681268031804)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=71 (bloomFilter=true), to=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/.tmp/cf1/1373dcc8adeb4ae4b9929ef46fbb4980 2023-04-12 02:55:37,943 INFO [rs(jenkins-hbase4.apache.org,43091,1681268031804)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=71 (bloomFilter=true), to=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/.tmp/cf2/a7d4784e93134ef2943643862fd08463 2023-04-12 02:55:37,948 DEBUG [rs(jenkins-hbase4.apache.org,43091,1681268031804)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/.tmp/cf1/1373dcc8adeb4ae4b9929ef46fbb4980 as hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf1/1373dcc8adeb4ae4b9929ef46fbb4980 2023-04-12 02:55:37,951 INFO [rs(jenkins-hbase4.apache.org,43091,1681268031804)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf1/1373dcc8adeb4ae4b9929ef46fbb4980, entries=10, sequenceid=71, filesize=5.0 K 2023-04-12 02:55:37,952 DEBUG [rs(jenkins-hbase4.apache.org,43091,1681268031804)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/.tmp/cf2/a7d4784e93134ef2943643862fd08463 as hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf2/a7d4784e93134ef2943643862fd08463 2023-04-12 02:55:37,957 INFO [rs(jenkins-hbase4.apache.org,43091,1681268031804)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf2/a7d4784e93134ef2943643862fd08463, entries=10, sequenceid=71, filesize=5.0 K 2023-04-12 02:55:37,957 INFO [rs(jenkins-hbase4.apache.org,43091,1681268031804)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for c09b5f7c44d4a83602090cbccc082b61 in 55ms, sequenceid=71, compaction requested=true 2023-04-12 02:55:37,958 DEBUG [rs(jenkins-hbase4.apache.org,43091,1681268031804)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c09b5f7c44d4a83602090cbccc082b61: 2023-04-12 02:55:37,958 DEBUG [rs(jenkins-hbase4.apache.org,43091,1681268031804)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testRecoveryAndDoubleExecution,,1681268136320.c09b5f7c44d4a83602090cbccc082b61. 2023-04-12 02:55:37,958 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 02:55:37,958 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 02:55:37,958 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:37,958 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testRecoveryAndDoubleExecution' locally acquired 2023-04-12 02:55:37,958 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43091,1681268031804' joining acquired barrier for procedure (testRecoveryAndDoubleExecution) in zk 2023-04-12 02:55:37,960 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:37,960 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:37,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 02:55:37,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 02:55:37,960 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,960 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testRecoveryAndDoubleExecution' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 02:55:37,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 02:55:37,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 02:55:37,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRecoveryAndDoubleExecution 2023-04-12 02:55:37,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:37,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:37,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:37,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 02:55:37,962 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43091,1681268031804' joining acquired barrier for procedure 'testRecoveryAndDoubleExecution' on coordinator 2023-04-12 02:55:37,962 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4b0a4a04[Count = 0] remaining members to acquire global barrier 2023-04-12 02:55:37,962 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testRecoveryAndDoubleExecution' starting 'in-barrier' execution. 2023-04-12 02:55:37,962 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,965 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,965 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,965 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,965 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,965 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,965 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testRecoveryAndDoubleExecution' received 'reached' from coordinator. 2023-04-12 02:55:37,965 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testRecoveryAndDoubleExecution' locally completed 2023-04-12 02:55:37,965 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testRecoveryAndDoubleExecution' received 'reached' from coordinator. 2023-04-12 02:55:37,965 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testRecoveryAndDoubleExecution' completed for member 'jenkins-hbase4.apache.org,34579,1681268031719' in zk 2023-04-12 02:55:37,965 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,965 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testRecoveryAndDoubleExecution' locally completed 2023-04-12 02:55:37,965 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testRecoveryAndDoubleExecution' received 'reached' from coordinator. 2023-04-12 02:55:37,965 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testRecoveryAndDoubleExecution' locally completed 2023-04-12 02:55:37,965 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testRecoveryAndDoubleExecution' completed for member 'jenkins-hbase4.apache.org,44679,1681268031772' in zk 2023-04-12 02:55:37,965 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testRecoveryAndDoubleExecution' completed for member 'jenkins-hbase4.apache.org,43091,1681268031804' in zk 2023-04-12 02:55:37,966 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:37,966 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 02:55:37,966 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testRecoveryAndDoubleExecution' has notified controller of completion 2023-04-12 02:55:37,966 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:55:37,966 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testRecoveryAndDoubleExecution' completed. 2023-04-12 02:55:37,967 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testRecoveryAndDoubleExecution' has notified controller of completion 2023-04-12 02:55:37,968 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:37,967 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testRecoveryAndDoubleExecution' has notified controller of completion 2023-04-12 02:55:37,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:37,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 02:55:37,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 02:55:37,968 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:55:37,968 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:55:37,968 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testRecoveryAndDoubleExecution' completed. 2023-04-12 02:55:37,968 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testRecoveryAndDoubleExecution' completed. 2023-04-12 02:55:37,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 02:55:37,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 02:55:37,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRecoveryAndDoubleExecution 2023-04-12 02:55:37,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:37,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:37,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:37,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 02:55:37,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRecoveryAndDoubleExecution 2023-04-12 02:55:37,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:37,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:37,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:37,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testRecoveryAndDoubleExecution' member 'jenkins-hbase4.apache.org,43091,1681268031804': 2023-04-12 02:55:37,972 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43091,1681268031804' released barrier for procedure'testRecoveryAndDoubleExecution', counting down latch. Waiting for 0 more 2023-04-12 02:55:37,972 INFO [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testRecoveryAndDoubleExecution' execution completed 2023-04-12 02:55:37,972 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 02:55:37,972 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 02:55:37,972 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testRecoveryAndDoubleExecution 2023-04-12 02:55:37,972 INFO [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testRecoveryAndDoubleExecutionincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 02:55:37,973 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,974 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,974 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,974 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,974 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,974 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:55:37,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 02:55:37,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 02:55:37,974 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,974 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:55:37,974 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:37,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,974 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,974 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:55:37,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 02:55:37,974 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:37,975 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:55:37,975 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:55:37,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:55:37,975 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:55:37,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:55:37,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRecoveryAndDoubleExecution 2023-04-12 02:55:37,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:55:37,975 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:37,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 02:55:37,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRecoveryAndDoubleExecution 2023-04-12 02:55:37,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:37,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:37,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:37,977 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:37,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 02:55:37,978 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:37,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRecoveryAndDoubleExecution 2023-04-12 02:55:37,978 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:37,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:37,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:37,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:37,984 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:37,984 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,984 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:37,984 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:55:37,984 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:55:37,984 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:55:37,984 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:55:37,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:55:37,984 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:55:37,984 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:55:37,984 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:55:37,985 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:55:37,985 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:55:37,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:55:37,985 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:55:37,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:55:37,985 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:55:37,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:55:37,984 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:37,984 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testRecoveryAndDoubleExecution' 2023-04-12 02:55:37,985 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 02:55:37,984 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:55:37,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:55:37,985 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,986 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:37,986 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,986 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:37,986 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:37,986 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,986 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-12 02:55:37,986 DEBUG [Listener at localhost/46667] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testRecoveryAndDoubleExecution'' to complete. (max 20000 ms per retry) 2023-04-12 02:55:37,986 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:55:37,986 DEBUG [Listener at localhost/46667] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 02:55:37,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:55:38,086 DEBUG [Listener at localhost/46667] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 02:55:38,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 02:55:38,346 WARN [Listener at localhost/46667] procedure2.ProcedureTestingUtility(193): Set Kill before store update to: true 2023-04-12 02:55:38,347 DEBUG [Listener at localhost/46667] assignment.SplitTableRegionProcedure(219): Splittable=true state=OPEN, location=jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:38,348 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(1029): Stored pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=c09b5f7c44d4a83602090cbccc082b61, daughterA=25b84f3651d3c0adac7c0275b4cb1e0c, daughterB=205d5359b7ecb778c3d784ef1bb72d34 2023-04-12 02:55:38,349 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=c09b5f7c44d4a83602090cbccc082b61, daughterA=25b84f3651d3c0adac7c0275b4cb1e0c, daughterB=205d5359b7ecb778c3d784ef1bb72d34 2023-04-12 02:55:38,349 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=c09b5f7c44d4a83602090cbccc082b61, daughterA=25b84f3651d3c0adac7c0275b4cb1e0c, daughterB=205d5359b7ecb778c3d784ef1bb72d34 2023-04-12 02:55:38,349 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=c09b5f7c44d4a83602090cbccc082b61, daughterA=25b84f3651d3c0adac7c0275b4cb1e0c, daughterB=205d5359b7ecb778c3d784ef1bb72d34 2023-04-12 02:55:38,350 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-04-12 02:55:38,350 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION, locked=true; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=c09b5f7c44d4a83602090cbccc082b61, daughterA=25b84f3651d3c0adac7c0275b4cb1e0c, daughterB=205d5359b7ecb778c3d784ef1bb72d34 2023-04-12 02:55:38,350 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-04-12 02:55:38,350 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION, locked=true; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=c09b5f7c44d4a83602090cbccc082b61, daughterA=25b84f3651d3c0adac7c0275b4cb1e0c, daughterB=205d5359b7ecb778c3d784ef1bb72d34 at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-04-12 02:55:38,598 INFO [Listener at localhost/46667] procedure.MasterProcedureTestingUtility(454): Restart 0 exec state=pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION, locked=true; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=c09b5f7c44d4a83602090cbccc082b61, daughterA=25b84f3651d3c0adac7c0275b4cb1e0c, daughterB=205d5359b7ecb778c3d784ef1bb72d34 2023-04-12 02:55:38,598 INFO [Listener at localhost/46667] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-04-12 02:55:38,598 INFO [Listener at localhost/46667] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-04-12 02:55:38,598 INFO [Listener at localhost/46667] assignment.AssignmentManager(315): Stopping assignment manager 2023-04-12 02:55:38,599 ERROR [Listener at localhost/46667] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] Thread[IPC Client (1542168714) connection to localhost/127.0.0.1:44691 from jenkins,5,PEWorkerGroup] Thread[HFileArchiver-9,5,PEWorkerGroup] Thread[HFileArchiver-10,5,PEWorkerGroup] Thread[HFileArchiver-11,5,PEWorkerGroup] Thread[HFileArchiver-12,5,PEWorkerGroup] 2023-04-12 02:55:38,599 INFO [Listener at localhost/46667] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-04-12 02:55:38,599 INFO [Listener at localhost/46667] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-04-12 02:55:38,599 INFO [Listener at localhost/46667] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-04-12 02:55:38,599 INFO [Listener at localhost/46667] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-04-12 02:55:38,599 INFO [Listener at localhost/46667] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-04-12 02:55:38,601 DEBUG [Listener at localhost/46667] store.ProcedureTree(114): Procedure Procedure(pid=112, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[] 2023-04-12 02:55:38,601 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-04-12 02:55:38,602 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-04-12 02:55:38,602 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-04-12 02:55:38,602 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-04-12 02:55:38,602 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-12 02:55:38,602 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=cc9012ec0b656baa24d04368fbe026c4, daughterA=e2a9b8fe15a781798c0b49e48ff5aad7, daughterB=6f55b5d6a460bd8c7569dee22f049104 2023-04-12 02:55:38,603 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-12 02:55:38,603 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-12 02:55:38,603 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-04-12 02:55:38,603 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=28, state=SUCCESS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=315e13e868a7f68a3ce37802b2f6312d, daughterA=4e1092f3c00484f8f09c730861225d43, daughterB=291734c00d212fd203b04cf8872d4649 2023-04-12 02:55:38,603 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=35, state=SUCCESS; DisableTableProcedure table=testSplitWithoutPONR 2023-04-12 02:55:38,603 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=40, state=SUCCESS; DeleteTableProcedure table=testSplitWithoutPONR 2023-04-12 02:55:38,604 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=41, state=SUCCESS; CreateTableProcedure table=testRollbackAndDoubleExecution 2023-04-12 02:55:38,604 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=44, state=ROLLEDBACK, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=96cdae2f7638212730b694522b9a464a, daughterA=fdd18e9cf539c13d863267ce16031583, daughterB=578095173fbc35e181cce14464037c2e 2023-04-12 02:55:38,604 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=47, state=SUCCESS; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=96cdae2f7638212730b694522b9a464a, ASSIGN 2023-04-12 02:55:38,604 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=49, state=SUCCESS; DisableTableProcedure table=testRollbackAndDoubleExecution 2023-04-12 02:55:38,604 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=52, state=SUCCESS; DeleteTableProcedure table=testRollbackAndDoubleExecution 2023-04-12 02:55:38,605 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=53, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionNoStoreFile 2023-04-12 02:55:38,605 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=56, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionNoStoreFile, parent=947433d660677a65ad9876ce2025e3e0, daughterA=b24cea6994eaa39f3128573197d02d9b, daughterB=b0e3d5fd7da33cb33084791956f0d7ca 2023-04-12 02:55:38,605 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=63, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionNoStoreFile 2023-04-12 02:55:38,605 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=68, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionNoStoreFile 2023-04-12 02:55:38,605 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=69, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:55:38,605 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=72, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionDeletedRowsDaughter, parent=1c6b8f18205322cbb248ea4fb10155d8, daughterA=81296e35c0d27ffa1ba7676ff53f87e1, daughterB=72b78dfd0678a920ddc3700612ff3932 2023-04-12 02:55:38,606 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=79, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:55:38,606 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=84, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:55:38,606 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=85, state=SUCCESS; CreateTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-04-12 02:55:38,606 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=88, state=SUCCESS; ModifyTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-04-12 02:55:38,607 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=95, state=ROLLEDBACK, exception=org.apache.hadoop.hbase.HBaseIOException via master-split-regions:org.apache.hadoop.hbase.HBaseIOException: The parent region state=OPENING, location=jenkins-hbase4.apache.org,34579,1681268031719, table=testRollbackForSplitTableRegionWithReplica, region=94855f8e18950d1c0c1e1d85df03f9ad is currently in transition, give up; SplitTableRegionProcedure table=testRollbackForSplitTableRegionWithReplica, parent=4381c9c8e57c5dbb2533908abaf07cf1, daughterA=2e35c2f6b8767f438f484c700a605299, daughterB=8407ddb6fc7704d121cd7f8a674aefb6 2023-04-12 02:55:38,607 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=96, state=SUCCESS; DisableTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-04-12 02:55:38,607 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=101, state=SUCCESS; DeleteTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-04-12 02:55:38,607 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=102, state=SUCCESS; CreateTableProcedure table=testInvalidSplitKey 2023-04-12 02:55:38,607 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=105, state=SUCCESS; DisableTableProcedure table=testInvalidSplitKey 2023-04-12 02:55:38,607 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=108, state=SUCCESS; DeleteTableProcedure table=testInvalidSplitKey 2023-04-12 02:55:38,608 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=109, state=SUCCESS; CreateTableProcedure table=testRecoveryAndDoubleExecution 2023-04-12 02:55:38,608 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(468): Loading pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=c09b5f7c44d4a83602090cbccc082b61, daughterA=25b84f3651d3c0adac7c0275b4cb1e0c, daughterB=205d5359b7ecb778c3d784ef1bb72d34 2023-04-12 02:55:38,608 DEBUG [Listener at localhost/46667] procedure2.Procedure(966): pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=c09b5f7c44d4a83602090cbccc082b61, daughterA=25b84f3651d3c0adac7c0275b4cb1e0c, daughterB=205d5359b7ecb778c3d784ef1bb72d34 held the lock before restarting, call acquireLock to restore it. 2023-04-12 02:55:38,608 INFO [Listener at localhost/46667] procedure.MasterProcedureScheduler(727): Took xlock for pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=c09b5f7c44d4a83602090cbccc082b61, daughterA=25b84f3651d3c0adac7c0275b4cb1e0c, daughterB=205d5359b7ecb778c3d784ef1bb72d34 2023-04-12 02:55:38,608 INFO [Listener at localhost/46667] procedure.MasterProcedureScheduler(727): Took xlock for pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=c09b5f7c44d4a83602090cbccc082b61, daughterA=25b84f3651d3c0adac7c0275b4cb1e0c, daughterB=205d5359b7ecb778c3d784ef1bb72d34 2023-04-12 02:55:38,608 INFO [Listener at localhost/46667] procedure.MasterProcedureScheduler(727): Took xlock for pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=c09b5f7c44d4a83602090cbccc082b61, daughterA=25b84f3651d3c0adac7c0275b4cb1e0c, daughterB=205d5359b7ecb778c3d784ef1bb72d34 2023-04-12 02:55:38,608 INFO [Listener at localhost/46667] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 8 msec 2023-04-12 02:55:38,609 DEBUG [Listener at localhost/46667] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-04-12 02:55:38,609 DEBUG [Listener at localhost/46667] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase4.apache.org,44679,1681268031772, table=hbase:meta, region=1588230740 2023-04-12 02:55:38,609 INFO [Listener at localhost/46667] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1681268168609 2023-04-12 02:55:38,610 DEBUG [Listener at localhost/46667] assignment.AssignmentManager(1517): Joining cluster... 2023-04-12 02:55:38,611 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=c09b5f7c44d4a83602090cbccc082b61, daughterA=25b84f3651d3c0adac7c0275b4cb1e0c, daughterB=205d5359b7ecb778c3d784ef1bb72d34 2023-04-12 02:55:38,614 DEBUG [Listener at localhost/46667] assignment.RegionStateStore(147): Load hbase:meta entry region=1f255e980a31bb266f2206a30aa2ae3f, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,44679,1681268031772, regionLocation=jenkins-hbase4.apache.org,44679,1681268031772, openSeqNum=2 2023-04-12 02:55:38,614 DEBUG [Listener at localhost/46667] assignment.RegionStateStore(147): Load hbase:meta entry region=c09b5f7c44d4a83602090cbccc082b61, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,43091,1681268031804, regionLocation=jenkins-hbase4.apache.org,43091,1681268031804, openSeqNum=2 2023-04-12 02:55:38,615 INFO [Listener at localhost/46667] assignment.AssignmentManager(1529): Number of RegionServers=3 2023-04-12 02:55:38,615 INFO [Listener at localhost/46667] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1681268198615 2023-04-12 02:55:38,615 INFO [Listener at localhost/46667] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1681268258615 2023-04-12 02:55:38,615 INFO [Listener at localhost/46667] assignment.AssignmentManager(1536): Joined the cluster in 4 msec 2023-04-12 02:55:38,615 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-04-12 02:55:38,617 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-04-12 02:55:38,617 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_CLOSE_PARENT_REGION, locked=true; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=c09b5f7c44d4a83602090cbccc082b61, daughterA=25b84f3651d3c0adac7c0275b4cb1e0c, daughterB=205d5359b7ecb778c3d784ef1bb72d34 2023-04-12 02:55:38,617 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-04-12 02:55:38,617 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_CLOSE_PARENT_REGION, locked=true; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=c09b5f7c44d4a83602090cbccc082b61, daughterA=25b84f3651d3c0adac7c0275b4cb1e0c, daughterB=205d5359b7ecb778c3d784ef1bb72d34 at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-04-12 02:55:38,865 INFO [Listener at localhost/46667] procedure.MasterProcedureTestingUtility(454): Restart 1 exec state=pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_CLOSE_PARENT_REGION, locked=true; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=c09b5f7c44d4a83602090cbccc082b61, daughterA=25b84f3651d3c0adac7c0275b4cb1e0c, daughterB=205d5359b7ecb778c3d784ef1bb72d34 2023-04-12 02:55:38,865 INFO [Listener at localhost/46667] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-04-12 02:55:38,865 INFO [Listener at localhost/46667] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-04-12 02:55:38,865 INFO [Listener at localhost/46667] assignment.AssignmentManager(315): Stopping assignment manager 2023-04-12 02:55:38,866 ERROR [Listener at localhost/46667] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-04-12 02:55:38,866 INFO [Listener at localhost/46667] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-04-12 02:55:38,866 INFO [Listener at localhost/46667] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-04-12 02:55:38,866 INFO [Listener at localhost/46667] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-04-12 02:55:38,866 INFO [Listener at localhost/46667] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-04-12 02:55:38,866 INFO [Listener at localhost/46667] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-04-12 02:55:38,868 DEBUG [Listener at localhost/46667] store.ProcedureTree(114): Procedure Procedure(pid=112, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[0] 2023-04-12 02:55:38,868 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-04-12 02:55:38,868 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-04-12 02:55:38,868 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-04-12 02:55:38,868 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-04-12 02:55:38,869 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-12 02:55:38,869 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=cc9012ec0b656baa24d04368fbe026c4, daughterA=e2a9b8fe15a781798c0b49e48ff5aad7, daughterB=6f55b5d6a460bd8c7569dee22f049104 2023-04-12 02:55:38,869 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-12 02:55:38,869 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-12 02:55:38,869 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-04-12 02:55:38,869 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=28, state=SUCCESS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=315e13e868a7f68a3ce37802b2f6312d, daughterA=4e1092f3c00484f8f09c730861225d43, daughterB=291734c00d212fd203b04cf8872d4649 2023-04-12 02:55:38,870 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=35, state=SUCCESS; DisableTableProcedure table=testSplitWithoutPONR 2023-04-12 02:55:38,870 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=40, state=SUCCESS; DeleteTableProcedure table=testSplitWithoutPONR 2023-04-12 02:55:38,870 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=41, state=SUCCESS; CreateTableProcedure table=testRollbackAndDoubleExecution 2023-04-12 02:55:38,870 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=44, state=ROLLEDBACK, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=96cdae2f7638212730b694522b9a464a, daughterA=fdd18e9cf539c13d863267ce16031583, daughterB=578095173fbc35e181cce14464037c2e 2023-04-12 02:55:38,870 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=47, state=SUCCESS; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=96cdae2f7638212730b694522b9a464a, ASSIGN 2023-04-12 02:55:38,870 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=49, state=SUCCESS; DisableTableProcedure table=testRollbackAndDoubleExecution 2023-04-12 02:55:38,870 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=52, state=SUCCESS; DeleteTableProcedure table=testRollbackAndDoubleExecution 2023-04-12 02:55:38,871 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=53, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionNoStoreFile 2023-04-12 02:55:38,871 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=56, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionNoStoreFile, parent=947433d660677a65ad9876ce2025e3e0, daughterA=b24cea6994eaa39f3128573197d02d9b, daughterB=b0e3d5fd7da33cb33084791956f0d7ca 2023-04-12 02:55:38,871 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=63, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionNoStoreFile 2023-04-12 02:55:38,871 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=68, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionNoStoreFile 2023-04-12 02:55:38,872 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=69, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:55:38,872 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=72, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionDeletedRowsDaughter, parent=1c6b8f18205322cbb248ea4fb10155d8, daughterA=81296e35c0d27ffa1ba7676ff53f87e1, daughterB=72b78dfd0678a920ddc3700612ff3932 2023-04-12 02:55:38,872 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=79, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:55:38,872 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=84, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:55:38,872 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=85, state=SUCCESS; CreateTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-04-12 02:55:38,872 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=88, state=SUCCESS; ModifyTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-04-12 02:55:38,873 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=95, state=ROLLEDBACK, exception=org.apache.hadoop.hbase.HBaseIOException via master-split-regions:org.apache.hadoop.hbase.HBaseIOException: The parent region state=OPENING, location=jenkins-hbase4.apache.org,34579,1681268031719, table=testRollbackForSplitTableRegionWithReplica, region=94855f8e18950d1c0c1e1d85df03f9ad is currently in transition, give up; SplitTableRegionProcedure table=testRollbackForSplitTableRegionWithReplica, parent=4381c9c8e57c5dbb2533908abaf07cf1, daughterA=2e35c2f6b8767f438f484c700a605299, daughterB=8407ddb6fc7704d121cd7f8a674aefb6 2023-04-12 02:55:38,873 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=96, state=SUCCESS; DisableTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-04-12 02:55:38,873 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=101, state=SUCCESS; DeleteTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-04-12 02:55:38,873 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=102, state=SUCCESS; CreateTableProcedure table=testInvalidSplitKey 2023-04-12 02:55:38,873 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=105, state=SUCCESS; DisableTableProcedure table=testInvalidSplitKey 2023-04-12 02:55:38,873 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=108, state=SUCCESS; DeleteTableProcedure table=testInvalidSplitKey 2023-04-12 02:55:38,873 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=109, state=SUCCESS; CreateTableProcedure table=testRecoveryAndDoubleExecution 2023-04-12 02:55:38,873 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(468): Loading pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=c09b5f7c44d4a83602090cbccc082b61, daughterA=25b84f3651d3c0adac7c0275b4cb1e0c, daughterB=205d5359b7ecb778c3d784ef1bb72d34 2023-04-12 02:55:38,873 DEBUG [Listener at localhost/46667] procedure2.Procedure(966): pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=c09b5f7c44d4a83602090cbccc082b61, daughterA=25b84f3651d3c0adac7c0275b4cb1e0c, daughterB=205d5359b7ecb778c3d784ef1bb72d34 held the lock before restarting, call acquireLock to restore it. 2023-04-12 02:55:38,874 INFO [Listener at localhost/46667] procedure.MasterProcedureScheduler(727): Took xlock for pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=c09b5f7c44d4a83602090cbccc082b61, daughterA=25b84f3651d3c0adac7c0275b4cb1e0c, daughterB=205d5359b7ecb778c3d784ef1bb72d34 2023-04-12 02:55:38,874 INFO [Listener at localhost/46667] procedure.MasterProcedureScheduler(727): Took xlock for pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=c09b5f7c44d4a83602090cbccc082b61, daughterA=25b84f3651d3c0adac7c0275b4cb1e0c, daughterB=205d5359b7ecb778c3d784ef1bb72d34 2023-04-12 02:55:38,874 INFO [Listener at localhost/46667] procedure.MasterProcedureScheduler(727): Took xlock for pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=c09b5f7c44d4a83602090cbccc082b61, daughterA=25b84f3651d3c0adac7c0275b4cb1e0c, daughterB=205d5359b7ecb778c3d784ef1bb72d34 2023-04-12 02:55:38,874 INFO [Listener at localhost/46667] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 7 msec 2023-04-12 02:55:38,874 DEBUG [Listener at localhost/46667] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-04-12 02:55:38,875 DEBUG [Listener at localhost/46667] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase4.apache.org,44679,1681268031772, table=hbase:meta, region=1588230740 2023-04-12 02:55:38,875 INFO [Listener at localhost/46667] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1681268168875 2023-04-12 02:55:38,875 DEBUG [Listener at localhost/46667] assignment.AssignmentManager(1517): Joining cluster... 2023-04-12 02:55:38,877 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=c09b5f7c44d4a83602090cbccc082b61, daughterA=25b84f3651d3c0adac7c0275b4cb1e0c, daughterB=205d5359b7ecb778c3d784ef1bb72d34 2023-04-12 02:55:38,879 DEBUG [Listener at localhost/46667] assignment.RegionStateStore(147): Load hbase:meta entry region=1f255e980a31bb266f2206a30aa2ae3f, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,44679,1681268031772, regionLocation=jenkins-hbase4.apache.org,44679,1681268031772, openSeqNum=2 2023-04-12 02:55:38,879 DEBUG [Listener at localhost/46667] assignment.RegionStateStore(147): Load hbase:meta entry region=c09b5f7c44d4a83602090cbccc082b61, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,43091,1681268031804, regionLocation=jenkins-hbase4.apache.org,43091,1681268031804, openSeqNum=2 2023-04-12 02:55:38,880 INFO [Listener at localhost/46667] assignment.AssignmentManager(1529): Number of RegionServers=3 2023-04-12 02:55:38,880 INFO [Listener at localhost/46667] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1681268198880 2023-04-12 02:55:38,880 INFO [Listener at localhost/46667] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1681268258880 2023-04-12 02:55:38,880 INFO [Listener at localhost/46667] assignment.AssignmentManager(1536): Joined the cluster in 4 msec 2023-04-12 02:55:38,880 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-04-12 02:55:38,882 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=113, ppid=112, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=c09b5f7c44d4a83602090cbccc082b61, UNASSIGN}] 2023-04-12 02:55:38,883 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-04-12 02:55:38,883 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=112, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS, locked=true; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=c09b5f7c44d4a83602090cbccc082b61, daughterA=25b84f3651d3c0adac7c0275b4cb1e0c, daughterB=205d5359b7ecb778c3d784ef1bb72d34 2023-04-12 02:55:38,883 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-04-12 02:55:38,883 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=112, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS, locked=true; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=c09b5f7c44d4a83602090cbccc082b61, daughterA=25b84f3651d3c0adac7c0275b4cb1e0c, daughterB=205d5359b7ecb778c3d784ef1bb72d34 at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-04-12 02:55:39,130 INFO [Listener at localhost/46667] procedure.MasterProcedureTestingUtility(454): Restart 2 exec state=pid=112, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS, locked=true; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=c09b5f7c44d4a83602090cbccc082b61, daughterA=25b84f3651d3c0adac7c0275b4cb1e0c, daughterB=205d5359b7ecb778c3d784ef1bb72d34 2023-04-12 02:55:39,130 INFO [Listener at localhost/46667] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-04-12 02:55:39,130 INFO [Listener at localhost/46667] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-04-12 02:55:39,130 INFO [Listener at localhost/46667] assignment.AssignmentManager(315): Stopping assignment manager 2023-04-12 02:55:39,131 ERROR [Listener at localhost/46667] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-04-12 02:55:39,131 INFO [Listener at localhost/46667] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-04-12 02:55:39,131 INFO [Listener at localhost/46667] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-04-12 02:55:39,131 INFO [Listener at localhost/46667] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-04-12 02:55:39,131 INFO [Listener at localhost/46667] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-04-12 02:55:39,131 INFO [Listener at localhost/46667] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-04-12 02:55:39,133 DEBUG [Listener at localhost/46667] store.ProcedureTree(114): Procedure Procedure(pid=112, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[0, 1] 2023-04-12 02:55:39,133 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-04-12 02:55:39,134 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-04-12 02:55:39,134 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-04-12 02:55:39,134 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-04-12 02:55:39,135 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-12 02:55:39,135 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=cc9012ec0b656baa24d04368fbe026c4, daughterA=e2a9b8fe15a781798c0b49e48ff5aad7, daughterB=6f55b5d6a460bd8c7569dee22f049104 2023-04-12 02:55:39,135 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-12 02:55:39,135 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-12 02:55:39,135 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-04-12 02:55:39,136 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=28, state=SUCCESS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=315e13e868a7f68a3ce37802b2f6312d, daughterA=4e1092f3c00484f8f09c730861225d43, daughterB=291734c00d212fd203b04cf8872d4649 2023-04-12 02:55:39,136 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=35, state=SUCCESS; DisableTableProcedure table=testSplitWithoutPONR 2023-04-12 02:55:39,136 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=40, state=SUCCESS; DeleteTableProcedure table=testSplitWithoutPONR 2023-04-12 02:55:39,136 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=41, state=SUCCESS; CreateTableProcedure table=testRollbackAndDoubleExecution 2023-04-12 02:55:39,137 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=44, state=ROLLEDBACK, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=96cdae2f7638212730b694522b9a464a, daughterA=fdd18e9cf539c13d863267ce16031583, daughterB=578095173fbc35e181cce14464037c2e 2023-04-12 02:55:39,137 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=47, state=SUCCESS; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=96cdae2f7638212730b694522b9a464a, ASSIGN 2023-04-12 02:55:39,137 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=49, state=SUCCESS; DisableTableProcedure table=testRollbackAndDoubleExecution 2023-04-12 02:55:39,137 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=52, state=SUCCESS; DeleteTableProcedure table=testRollbackAndDoubleExecution 2023-04-12 02:55:39,138 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=53, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionNoStoreFile 2023-04-12 02:55:39,138 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=56, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionNoStoreFile, parent=947433d660677a65ad9876ce2025e3e0, daughterA=b24cea6994eaa39f3128573197d02d9b, daughterB=b0e3d5fd7da33cb33084791956f0d7ca 2023-04-12 02:55:39,138 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=63, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionNoStoreFile 2023-04-12 02:55:39,138 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=68, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionNoStoreFile 2023-04-12 02:55:39,138 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=69, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:55:39,139 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=72, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionDeletedRowsDaughter, parent=1c6b8f18205322cbb248ea4fb10155d8, daughterA=81296e35c0d27ffa1ba7676ff53f87e1, daughterB=72b78dfd0678a920ddc3700612ff3932 2023-04-12 02:55:39,139 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=79, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:55:39,139 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=84, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:55:39,139 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=85, state=SUCCESS; CreateTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-04-12 02:55:39,140 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=88, state=SUCCESS; ModifyTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-04-12 02:55:39,140 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=95, state=ROLLEDBACK, exception=org.apache.hadoop.hbase.HBaseIOException via master-split-regions:org.apache.hadoop.hbase.HBaseIOException: The parent region state=OPENING, location=jenkins-hbase4.apache.org,34579,1681268031719, table=testRollbackForSplitTableRegionWithReplica, region=94855f8e18950d1c0c1e1d85df03f9ad is currently in transition, give up; SplitTableRegionProcedure table=testRollbackForSplitTableRegionWithReplica, parent=4381c9c8e57c5dbb2533908abaf07cf1, daughterA=2e35c2f6b8767f438f484c700a605299, daughterB=8407ddb6fc7704d121cd7f8a674aefb6 2023-04-12 02:55:39,140 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=96, state=SUCCESS; DisableTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-04-12 02:55:39,140 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=101, state=SUCCESS; DeleteTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-04-12 02:55:39,141 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=102, state=SUCCESS; CreateTableProcedure table=testInvalidSplitKey 2023-04-12 02:55:39,141 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=105, state=SUCCESS; DisableTableProcedure table=testInvalidSplitKey 2023-04-12 02:55:39,141 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=108, state=SUCCESS; DeleteTableProcedure table=testInvalidSplitKey 2023-04-12 02:55:39,141 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=109, state=SUCCESS; CreateTableProcedure table=testRecoveryAndDoubleExecution 2023-04-12 02:55:39,141 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(468): Loading pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_CLOSE_PARENT_REGION; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=c09b5f7c44d4a83602090cbccc082b61, daughterA=25b84f3651d3c0adac7c0275b4cb1e0c, daughterB=205d5359b7ecb778c3d784ef1bb72d34 2023-04-12 02:55:39,142 DEBUG [Listener at localhost/46667] procedure2.Procedure(966): pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_CLOSE_PARENT_REGION; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=c09b5f7c44d4a83602090cbccc082b61, daughterA=25b84f3651d3c0adac7c0275b4cb1e0c, daughterB=205d5359b7ecb778c3d784ef1bb72d34 held the lock before restarting, call acquireLock to restore it. 2023-04-12 02:55:39,142 INFO [Listener at localhost/46667] procedure.MasterProcedureScheduler(727): Took xlock for pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_CLOSE_PARENT_REGION; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=c09b5f7c44d4a83602090cbccc082b61, daughterA=25b84f3651d3c0adac7c0275b4cb1e0c, daughterB=205d5359b7ecb778c3d784ef1bb72d34 2023-04-12 02:55:39,142 INFO [Listener at localhost/46667] procedure.MasterProcedureScheduler(727): Took xlock for pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_CLOSE_PARENT_REGION; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=c09b5f7c44d4a83602090cbccc082b61, daughterA=25b84f3651d3c0adac7c0275b4cb1e0c, daughterB=205d5359b7ecb778c3d784ef1bb72d34 2023-04-12 02:55:39,142 INFO [Listener at localhost/46667] procedure.MasterProcedureScheduler(727): Took xlock for pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_CLOSE_PARENT_REGION; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=c09b5f7c44d4a83602090cbccc082b61, daughterA=25b84f3651d3c0adac7c0275b4cb1e0c, daughterB=205d5359b7ecb778c3d784ef1bb72d34 2023-04-12 02:55:39,142 INFO [Listener at localhost/46667] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 10 msec 2023-04-12 02:55:39,142 DEBUG [Listener at localhost/46667] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-04-12 02:55:39,143 DEBUG [Listener at localhost/46667] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase4.apache.org,44679,1681268031772, table=hbase:meta, region=1588230740 2023-04-12 02:55:39,146 INFO [Listener at localhost/46667] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1681268169146 2023-04-12 02:55:39,147 DEBUG [Listener at localhost/46667] assignment.AssignmentManager(1517): Joining cluster... 2023-04-12 02:55:39,148 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_CLOSE_PARENT_REGION; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=c09b5f7c44d4a83602090cbccc082b61, daughterA=25b84f3651d3c0adac7c0275b4cb1e0c, daughterB=205d5359b7ecb778c3d784ef1bb72d34 2023-04-12 02:55:39,150 DEBUG [Listener at localhost/46667] assignment.RegionStateStore(147): Load hbase:meta entry region=1f255e980a31bb266f2206a30aa2ae3f, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,44679,1681268031772, regionLocation=jenkins-hbase4.apache.org,44679,1681268031772, openSeqNum=2 2023-04-12 02:55:39,151 DEBUG [Listener at localhost/46667] assignment.RegionStateStore(147): Load hbase:meta entry region=c09b5f7c44d4a83602090cbccc082b61, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,43091,1681268031804, regionLocation=jenkins-hbase4.apache.org,43091,1681268031804, openSeqNum=2 2023-04-12 02:55:39,151 INFO [Listener at localhost/46667] assignment.AssignmentManager(1529): Number of RegionServers=3 2023-04-12 02:55:39,151 INFO [Listener at localhost/46667] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1681268199151 2023-04-12 02:55:39,151 INFO [Listener at localhost/46667] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1681268259151 2023-04-12 02:55:39,151 INFO [Listener at localhost/46667] assignment.AssignmentManager(1536): Joined the cluster in 3 msec 2023-04-12 02:55:39,151 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=113, ppid=112, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=c09b5f7c44d4a83602090cbccc082b61, UNASSIGN}] 2023-04-12 02:55:39,152 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-04-12 02:55:39,154 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=113, ppid=112, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=c09b5f7c44d4a83602090cbccc082b61, UNASSIGN 2023-04-12 02:55:39,155 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=113 updating hbase:meta row=c09b5f7c44d4a83602090cbccc082b61, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:39,155 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRecoveryAndDoubleExecution,,1681268136320.c09b5f7c44d4a83602090cbccc082b61.","families":{"info":[{"qualifier":"regioninfo","vlen":64,"tag":[],"timestamp":"1681268139155"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681268139155"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681268139155"}]},"ts":"1681268139155"} 2023-04-12 02:55:39,156 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=114, ppid=113, state=RUNNABLE; CloseRegionProcedure c09b5f7c44d4a83602090cbccc082b61, server=jenkins-hbase4.apache.org,43091,1681268031804}] 2023-04-12 02:55:39,262 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-12 02:55:39,308 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close c09b5f7c44d4a83602090cbccc082b61 2023-04-12 02:55:39,308 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing c09b5f7c44d4a83602090cbccc082b61, disabling compactions & flushes 2023-04-12 02:55:39,308 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region testRecoveryAndDoubleExecution,,1681268136320.c09b5f7c44d4a83602090cbccc082b61. 2023-04-12 02:55:39,308 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testRecoveryAndDoubleExecution,,1681268136320.c09b5f7c44d4a83602090cbccc082b61. 2023-04-12 02:55:39,308 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on testRecoveryAndDoubleExecution,,1681268136320.c09b5f7c44d4a83602090cbccc082b61. after waiting 0 ms 2023-04-12 02:55:39,308 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region testRecoveryAndDoubleExecution,,1681268136320.c09b5f7c44d4a83602090cbccc082b61. 2023-04-12 02:55:39,309 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing c09b5f7c44d4a83602090cbccc082b61 2/2 column families, dataSize=480 B heapSize=2.13 KB 2023-04-12 02:55:39,328 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=240 B at sequenceid=82 (bloomFilter=true), to=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/.tmp/cf1/e62dda4fe11148e88206bdd397ca64c0 2023-04-12 02:55:39,351 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=240 B at sequenceid=82 (bloomFilter=true), to=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/.tmp/cf2/f70a7ebc6c464a08ab00a25e137bbc72 2023-04-12 02:55:39,355 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/.tmp/cf1/e62dda4fe11148e88206bdd397ca64c0 as hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf1/e62dda4fe11148e88206bdd397ca64c0 2023-04-12 02:55:39,359 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf1/e62dda4fe11148e88206bdd397ca64c0, entries=8, sequenceid=82, filesize=5.0 K 2023-04-12 02:55:39,359 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/.tmp/cf2/f70a7ebc6c464a08ab00a25e137bbc72 as hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf2/f70a7ebc6c464a08ab00a25e137bbc72 2023-04-12 02:55:39,364 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf2/f70a7ebc6c464a08ab00a25e137bbc72, entries=8, sequenceid=82, filesize=5.0 K 2023-04-12 02:55:39,365 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~480 B/480, heapSize ~2.09 KB/2144, currentSize=0 B/0 for c09b5f7c44d4a83602090cbccc082b61 in 56ms, sequenceid=82, compaction requested=true 2023-04-12 02:55:39,369 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/recovered.edits/85.seqid, newMaxSeqId=85, maxSeqId=1 2023-04-12 02:55:39,370 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro 2023-04-12 02:55:39,371 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed testRecoveryAndDoubleExecution,,1681268136320.c09b5f7c44d4a83602090cbccc082b61. 2023-04-12 02:55:39,371 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for c09b5f7c44d4a83602090cbccc082b61: 2023-04-12 02:55:39,373 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed c09b5f7c44d4a83602090cbccc082b61 2023-04-12 02:55:39,373 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=113 updating hbase:meta row=c09b5f7c44d4a83602090cbccc082b61, regionState=CLOSED 2023-04-12 02:55:39,374 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRecoveryAndDoubleExecution,,1681268136320.c09b5f7c44d4a83602090cbccc082b61.","families":{"info":[{"qualifier":"regioninfo","vlen":64,"tag":[],"timestamp":"1681268139373"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681268139373"}]},"ts":"1681268139373"} 2023-04-12 02:55:39,377 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=114, resume processing ppid=113 2023-04-12 02:55:39,377 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=114, ppid=113, state=SUCCESS; CloseRegionProcedure c09b5f7c44d4a83602090cbccc082b61, server=jenkins-hbase4.apache.org,43091,1681268031804 in 219 msec 2023-04-12 02:55:39,379 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=113, resume processing ppid=112 2023-04-12 02:55:39,379 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=113, ppid=112, state=SUCCESS; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=c09b5f7c44d4a83602090cbccc082b61, UNASSIGN in 227 msec 2023-04-12 02:55:39,381 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-04-12 02:55:39,381 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_CREATE_DAUGHTER_REGIONS, locked=true; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=c09b5f7c44d4a83602090cbccc082b61, daughterA=25b84f3651d3c0adac7c0275b4cb1e0c, daughterB=205d5359b7ecb778c3d784ef1bb72d34 2023-04-12 02:55:39,381 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-04-12 02:55:39,381 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_CREATE_DAUGHTER_REGIONS, locked=true; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=c09b5f7c44d4a83602090cbccc082b61, daughterA=25b84f3651d3c0adac7c0275b4cb1e0c, daughterB=205d5359b7ecb778c3d784ef1bb72d34 at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-04-12 02:55:39,401 INFO [Listener at localhost/46667] procedure.MasterProcedureTestingUtility(454): Restart 3 exec state=pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_CREATE_DAUGHTER_REGIONS, locked=true; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=c09b5f7c44d4a83602090cbccc082b61, daughterA=25b84f3651d3c0adac7c0275b4cb1e0c, daughterB=205d5359b7ecb778c3d784ef1bb72d34 2023-04-12 02:55:39,401 INFO [Listener at localhost/46667] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-04-12 02:55:39,402 INFO [Listener at localhost/46667] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-04-12 02:55:39,402 INFO [Listener at localhost/46667] assignment.AssignmentManager(315): Stopping assignment manager 2023-04-12 02:55:39,402 ERROR [Listener at localhost/46667] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-04-12 02:55:39,402 INFO [Listener at localhost/46667] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-04-12 02:55:39,402 INFO [Listener at localhost/46667] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-04-12 02:55:39,402 INFO [Listener at localhost/46667] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-04-12 02:55:39,402 INFO [Listener at localhost/46667] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-04-12 02:55:39,403 INFO [Listener at localhost/46667] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-04-12 02:55:39,404 DEBUG [Listener at localhost/46667] store.ProcedureTree(114): Procedure Procedure(pid=112, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[0, 1, 2] 2023-04-12 02:55:39,405 DEBUG [Listener at localhost/46667] store.ProcedureTree(114): Procedure Procedure(pid=113, ppid=112, class=org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure) stack ids=[3, 6] 2023-04-12 02:55:39,405 DEBUG [Listener at localhost/46667] store.ProcedureTree(114): Procedure Procedure(pid=114, ppid=113, class=org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure) stack ids=[4, 5] 2023-04-12 02:55:39,405 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-04-12 02:55:39,405 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-04-12 02:55:39,406 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-04-12 02:55:39,406 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-04-12 02:55:39,406 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-12 02:55:39,406 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=cc9012ec0b656baa24d04368fbe026c4, daughterA=e2a9b8fe15a781798c0b49e48ff5aad7, daughterB=6f55b5d6a460bd8c7569dee22f049104 2023-04-12 02:55:39,406 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-12 02:55:39,407 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-12 02:55:39,407 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-04-12 02:55:39,407 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=28, state=SUCCESS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=315e13e868a7f68a3ce37802b2f6312d, daughterA=4e1092f3c00484f8f09c730861225d43, daughterB=291734c00d212fd203b04cf8872d4649 2023-04-12 02:55:39,407 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=35, state=SUCCESS; DisableTableProcedure table=testSplitWithoutPONR 2023-04-12 02:55:39,407 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=40, state=SUCCESS; DeleteTableProcedure table=testSplitWithoutPONR 2023-04-12 02:55:39,407 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=41, state=SUCCESS; CreateTableProcedure table=testRollbackAndDoubleExecution 2023-04-12 02:55:39,408 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=44, state=ROLLEDBACK, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=96cdae2f7638212730b694522b9a464a, daughterA=fdd18e9cf539c13d863267ce16031583, daughterB=578095173fbc35e181cce14464037c2e 2023-04-12 02:55:39,408 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=47, state=SUCCESS; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=96cdae2f7638212730b694522b9a464a, ASSIGN 2023-04-12 02:55:39,408 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=49, state=SUCCESS; DisableTableProcedure table=testRollbackAndDoubleExecution 2023-04-12 02:55:39,408 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=52, state=SUCCESS; DeleteTableProcedure table=testRollbackAndDoubleExecution 2023-04-12 02:55:39,409 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=53, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionNoStoreFile 2023-04-12 02:55:39,409 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=56, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionNoStoreFile, parent=947433d660677a65ad9876ce2025e3e0, daughterA=b24cea6994eaa39f3128573197d02d9b, daughterB=b0e3d5fd7da33cb33084791956f0d7ca 2023-04-12 02:55:39,409 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=63, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionNoStoreFile 2023-04-12 02:55:39,409 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=68, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionNoStoreFile 2023-04-12 02:55:39,409 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=69, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:55:39,410 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=72, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionDeletedRowsDaughter, parent=1c6b8f18205322cbb248ea4fb10155d8, daughterA=81296e35c0d27ffa1ba7676ff53f87e1, daughterB=72b78dfd0678a920ddc3700612ff3932 2023-04-12 02:55:39,410 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=79, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:55:39,410 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=84, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:55:39,410 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=85, state=SUCCESS; CreateTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-04-12 02:55:39,411 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=88, state=SUCCESS; ModifyTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-04-12 02:55:39,411 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=95, state=ROLLEDBACK, exception=org.apache.hadoop.hbase.HBaseIOException via master-split-regions:org.apache.hadoop.hbase.HBaseIOException: The parent region state=OPENING, location=jenkins-hbase4.apache.org,34579,1681268031719, table=testRollbackForSplitTableRegionWithReplica, region=94855f8e18950d1c0c1e1d85df03f9ad is currently in transition, give up; SplitTableRegionProcedure table=testRollbackForSplitTableRegionWithReplica, parent=4381c9c8e57c5dbb2533908abaf07cf1, daughterA=2e35c2f6b8767f438f484c700a605299, daughterB=8407ddb6fc7704d121cd7f8a674aefb6 2023-04-12 02:55:39,411 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=96, state=SUCCESS; DisableTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-04-12 02:55:39,411 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=101, state=SUCCESS; DeleteTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-04-12 02:55:39,411 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=102, state=SUCCESS; CreateTableProcedure table=testInvalidSplitKey 2023-04-12 02:55:39,412 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=105, state=SUCCESS; DisableTableProcedure table=testInvalidSplitKey 2023-04-12 02:55:39,412 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=108, state=SUCCESS; DeleteTableProcedure table=testInvalidSplitKey 2023-04-12 02:55:39,412 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=109, state=SUCCESS; CreateTableProcedure table=testRecoveryAndDoubleExecution 2023-04-12 02:55:39,413 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(468): Loading pid=112, state=RUNNABLE:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=c09b5f7c44d4a83602090cbccc082b61, daughterA=25b84f3651d3c0adac7c0275b4cb1e0c, daughterB=205d5359b7ecb778c3d784ef1bb72d34 2023-04-12 02:55:39,413 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(468): Loading pid=113, ppid=112, state=SUCCESS; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=c09b5f7c44d4a83602090cbccc082b61, UNASSIGN 2023-04-12 02:55:39,413 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(468): Loading pid=114, ppid=113, state=SUCCESS; CloseRegionProcedure c09b5f7c44d4a83602090cbccc082b61, server=jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:39,413 DEBUG [Listener at localhost/46667] procedure2.Procedure(966): pid=112, state=RUNNABLE:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=c09b5f7c44d4a83602090cbccc082b61, daughterA=25b84f3651d3c0adac7c0275b4cb1e0c, daughterB=205d5359b7ecb778c3d784ef1bb72d34 held the lock before restarting, call acquireLock to restore it. 2023-04-12 02:55:39,413 INFO [Listener at localhost/46667] procedure.MasterProcedureScheduler(727): Took xlock for pid=112, state=RUNNABLE:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=c09b5f7c44d4a83602090cbccc082b61, daughterA=25b84f3651d3c0adac7c0275b4cb1e0c, daughterB=205d5359b7ecb778c3d784ef1bb72d34 2023-04-12 02:55:39,413 INFO [Listener at localhost/46667] procedure.MasterProcedureScheduler(727): Took xlock for pid=112, state=RUNNABLE:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=c09b5f7c44d4a83602090cbccc082b61, daughterA=25b84f3651d3c0adac7c0275b4cb1e0c, daughterB=205d5359b7ecb778c3d784ef1bb72d34 2023-04-12 02:55:39,413 INFO [Listener at localhost/46667] procedure.MasterProcedureScheduler(727): Took xlock for pid=112, state=RUNNABLE:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=c09b5f7c44d4a83602090cbccc082b61, daughterA=25b84f3651d3c0adac7c0275b4cb1e0c, daughterB=205d5359b7ecb778c3d784ef1bb72d34 2023-04-12 02:55:39,413 DEBUG [Listener at localhost/46667] procedure2.Procedure(946): pid=113, ppid=112, state=SUCCESS; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=c09b5f7c44d4a83602090cbccc082b61, UNASSIGN didn't hold the lock before restarting, skip acquiring lock. 2023-04-12 02:55:39,414 DEBUG [Listener at localhost/46667] procedure2.Procedure(946): pid=114, ppid=113, state=SUCCESS; CloseRegionProcedure c09b5f7c44d4a83602090cbccc082b61, server=jenkins-hbase4.apache.org,43091,1681268031804 didn't hold the lock before restarting, skip acquiring lock. 2023-04-12 02:55:39,414 INFO [Listener at localhost/46667] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 10 msec 2023-04-12 02:55:39,416 DEBUG [Listener at localhost/46667] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-04-12 02:55:39,417 DEBUG [Listener at localhost/46667] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase4.apache.org,44679,1681268031772, table=hbase:meta, region=1588230740 2023-04-12 02:55:39,418 INFO [Listener at localhost/46667] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1681268169418 2023-04-12 02:55:39,418 DEBUG [Listener at localhost/46667] assignment.AssignmentManager(1517): Joining cluster... 2023-04-12 02:55:39,421 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=112, state=RUNNABLE:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=c09b5f7c44d4a83602090cbccc082b61, daughterA=25b84f3651d3c0adac7c0275b4cb1e0c, daughterB=205d5359b7ecb778c3d784ef1bb72d34 2023-04-12 02:55:39,423 DEBUG [Listener at localhost/46667] assignment.RegionStateStore(147): Load hbase:meta entry region=1f255e980a31bb266f2206a30aa2ae3f, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,44679,1681268031772, regionLocation=jenkins-hbase4.apache.org,44679,1681268031772, openSeqNum=2 2023-04-12 02:55:39,423 DEBUG [Listener at localhost/46667] assignment.RegionStateStore(147): Load hbase:meta entry region=c09b5f7c44d4a83602090cbccc082b61, regionState=CLOSED, lastHost=jenkins-hbase4.apache.org,43091,1681268031804, regionLocation=jenkins-hbase4.apache.org,43091,1681268031804, openSeqNum=2 2023-04-12 02:55:39,423 INFO [Listener at localhost/46667] assignment.AssignmentManager(1529): Number of RegionServers=3 2023-04-12 02:55:39,424 INFO [Listener at localhost/46667] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1681268199424 2023-04-12 02:55:39,424 INFO [Listener at localhost/46667] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1681268259424 2023-04-12 02:55:39,424 INFO [Listener at localhost/46667] assignment.AssignmentManager(1536): Joined the cluster in 5 msec 2023-04-12 02:55:39,426 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-04-12 02:55:39,436 INFO [PEWorker-1] assignment.SplitTableRegionProcedure(694): pid=112 splitting 14 storefiles, region=c09b5f7c44d4a83602090cbccc082b61, threads=14 2023-04-12 02:55:39,436 DEBUG [StoreFileSplitter-pool-0] assignment.SplitTableRegionProcedure(776): pid=112 splitting started for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf2/1b3fc837d85449cba8b914aa7f18f0fe for region: c09b5f7c44d4a83602090cbccc082b61 2023-04-12 02:55:39,436 DEBUG [StoreFileSplitter-pool-1] assignment.SplitTableRegionProcedure(776): pid=112 splitting started for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf2/5b22fe6156a04c8c9486fd6f189bf4d7 for region: c09b5f7c44d4a83602090cbccc082b61 2023-04-12 02:55:39,437 DEBUG [StoreFileSplitter-pool-2] assignment.SplitTableRegionProcedure(776): pid=112 splitting started for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf2/a4e6a82328b24916838ccb970ddd7704 for region: c09b5f7c44d4a83602090cbccc082b61 2023-04-12 02:55:39,439 DEBUG [StoreFileSplitter-pool-3] assignment.SplitTableRegionProcedure(776): pid=112 splitting started for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf2/a7d4784e93134ef2943643862fd08463 for region: c09b5f7c44d4a83602090cbccc082b61 2023-04-12 02:55:39,440 DEBUG [StoreFileSplitter-pool-4] assignment.SplitTableRegionProcedure(776): pid=112 splitting started for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf2/f70a7ebc6c464a08ab00a25e137bbc72 for region: c09b5f7c44d4a83602090cbccc082b61 2023-04-12 02:55:39,444 DEBUG [StoreFileSplitter-pool-5] assignment.SplitTableRegionProcedure(776): pid=112 splitting started for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf2/fb8ce8dc74bd47fabd120f6f688962d5 for region: c09b5f7c44d4a83602090cbccc082b61 2023-04-12 02:55:39,445 DEBUG [StoreFileSplitter-pool-6] assignment.SplitTableRegionProcedure(776): pid=112 splitting started for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf2/fe343f29642942069fb8e38396f28358 for region: c09b5f7c44d4a83602090cbccc082b61 2023-04-12 02:55:39,448 DEBUG [StoreFileSplitter-pool-7] assignment.SplitTableRegionProcedure(776): pid=112 splitting started for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf1/001e134c60444ac2b388f9df6bf5670b for region: c09b5f7c44d4a83602090cbccc082b61 2023-04-12 02:55:39,449 DEBUG [StoreFileSplitter-pool-8] assignment.SplitTableRegionProcedure(776): pid=112 splitting started for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf1/1373dcc8adeb4ae4b9929ef46fbb4980 for region: c09b5f7c44d4a83602090cbccc082b61 2023-04-12 02:55:39,450 DEBUG [StoreFileSplitter-pool-9] assignment.SplitTableRegionProcedure(776): pid=112 splitting started for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf1/63025bd542924f2caced3f22c416a5a1 for region: c09b5f7c44d4a83602090cbccc082b61 2023-04-12 02:55:39,450 DEBUG [StoreFileSplitter-pool-10] assignment.SplitTableRegionProcedure(776): pid=112 splitting started for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf1/9d5ec6ededfd4de8b6d44e635d18346c for region: c09b5f7c44d4a83602090cbccc082b61 2023-04-12 02:55:39,450 DEBUG [StoreFileSplitter-pool-11] assignment.SplitTableRegionProcedure(776): pid=112 splitting started for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf1/a79c7cc5c59c4f6bbe6a6a584d824f36 for region: c09b5f7c44d4a83602090cbccc082b61 2023-04-12 02:55:39,450 DEBUG [StoreFileSplitter-pool-12] assignment.SplitTableRegionProcedure(776): pid=112 splitting started for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf1/c14bd2fe66004d6d842c72f4b2a4ee90 for region: c09b5f7c44d4a83602090cbccc082b61 2023-04-12 02:55:39,451 DEBUG [StoreFileSplitter-pool-13] assignment.SplitTableRegionProcedure(776): pid=112 splitting started for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf1/e62dda4fe11148e88206bdd397ca64c0 for region: c09b5f7c44d4a83602090cbccc082b61 2023-04-12 02:55:39,746 DEBUG [StoreFileSplitter-pool-1] assignment.SplitTableRegionProcedure(787): pid=112 splitting complete for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf2/5b22fe6156a04c8c9486fd6f189bf4d7 for region: c09b5f7c44d4a83602090cbccc082b61 2023-04-12 02:55:39,756 DEBUG [StoreFileSplitter-pool-8] assignment.SplitTableRegionProcedure(787): pid=112 splitting complete for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf1/1373dcc8adeb4ae4b9929ef46fbb4980 for region: c09b5f7c44d4a83602090cbccc082b61 2023-04-12 02:55:39,760 DEBUG [StoreFileSplitter-pool-11] assignment.SplitTableRegionProcedure(787): pid=112 splitting complete for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf1/a79c7cc5c59c4f6bbe6a6a584d824f36 for region: c09b5f7c44d4a83602090cbccc082b61 2023-04-12 02:55:39,772 DEBUG [StoreFileSplitter-pool-6] assignment.SplitTableRegionProcedure(787): pid=112 splitting complete for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf2/fe343f29642942069fb8e38396f28358 for region: c09b5f7c44d4a83602090cbccc082b61 2023-04-12 02:55:39,773 DEBUG [StoreFileSplitter-pool-9] assignment.SplitTableRegionProcedure(787): pid=112 splitting complete for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf1/63025bd542924f2caced3f22c416a5a1 for region: c09b5f7c44d4a83602090cbccc082b61 2023-04-12 02:55:39,783 DEBUG [StoreFileSplitter-pool-5] assignment.SplitTableRegionProcedure(787): pid=112 splitting complete for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf2/fb8ce8dc74bd47fabd120f6f688962d5 for region: c09b5f7c44d4a83602090cbccc082b61 2023-04-12 02:55:39,805 DEBUG [StoreFileSplitter-pool-10] assignment.SplitTableRegionProcedure(787): pid=112 splitting complete for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf1/9d5ec6ededfd4de8b6d44e635d18346c for region: c09b5f7c44d4a83602090cbccc082b61 2023-04-12 02:55:39,805 DEBUG [StoreFileSplitter-pool-7] assignment.SplitTableRegionProcedure(787): pid=112 splitting complete for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf1/001e134c60444ac2b388f9df6bf5670b for region: c09b5f7c44d4a83602090cbccc082b61 2023-04-12 02:55:39,887 DEBUG [StoreFileSplitter-pool-13] assignment.SplitTableRegionProcedure(787): pid=112 splitting complete for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf1/e62dda4fe11148e88206bdd397ca64c0 for region: c09b5f7c44d4a83602090cbccc082b61 2023-04-12 02:55:39,896 DEBUG [StoreFileSplitter-pool-2] assignment.SplitTableRegionProcedure(787): pid=112 splitting complete for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf2/a4e6a82328b24916838ccb970ddd7704 for region: c09b5f7c44d4a83602090cbccc082b61 2023-04-12 02:55:39,916 DEBUG [StoreFileSplitter-pool-3] assignment.SplitTableRegionProcedure(787): pid=112 splitting complete for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf2/a7d4784e93134ef2943643862fd08463 for region: c09b5f7c44d4a83602090cbccc082b61 2023-04-12 02:55:39,920 DEBUG [StoreFileSplitter-pool-4] assignment.SplitTableRegionProcedure(787): pid=112 splitting complete for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf2/f70a7ebc6c464a08ab00a25e137bbc72 for region: c09b5f7c44d4a83602090cbccc082b61 2023-04-12 02:55:39,927 DEBUG [StoreFileSplitter-pool-12] assignment.SplitTableRegionProcedure(787): pid=112 splitting complete for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf1/c14bd2fe66004d6d842c72f4b2a4ee90 for region: c09b5f7c44d4a83602090cbccc082b61 2023-04-12 02:55:40,257 DEBUG [StoreFileSplitter-pool-0] assignment.SplitTableRegionProcedure(787): pid=112 splitting complete for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf2/1b3fc837d85449cba8b914aa7f18f0fe for region: c09b5f7c44d4a83602090cbccc082b61 2023-04-12 02:55:40,257 DEBUG [PEWorker-1] assignment.SplitTableRegionProcedure(755): pid=112 split storefiles for region c09b5f7c44d4a83602090cbccc082b61 Daughter A: 14 storefiles, Daughter B: 14 storefiles. 2023-04-12 02:55:40,299 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-04-12 02:55:40,299 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_WRITE_MAX_SEQUENCE_ID_FILE, locked=true; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=c09b5f7c44d4a83602090cbccc082b61, daughterA=25b84f3651d3c0adac7c0275b4cb1e0c, daughterB=205d5359b7ecb778c3d784ef1bb72d34 2023-04-12 02:55:40,299 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-04-12 02:55:40,299 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_WRITE_MAX_SEQUENCE_ID_FILE, locked=true; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=c09b5f7c44d4a83602090cbccc082b61, daughterA=25b84f3651d3c0adac7c0275b4cb1e0c, daughterB=205d5359b7ecb778c3d784ef1bb72d34 at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-04-12 02:55:40,425 INFO [Listener at localhost/46667] procedure.MasterProcedureTestingUtility(454): Restart 4 exec state=pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_WRITE_MAX_SEQUENCE_ID_FILE, locked=true; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=c09b5f7c44d4a83602090cbccc082b61, daughterA=25b84f3651d3c0adac7c0275b4cb1e0c, daughterB=205d5359b7ecb778c3d784ef1bb72d34 2023-04-12 02:55:40,425 INFO [Listener at localhost/46667] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-04-12 02:55:40,425 INFO [Listener at localhost/46667] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-04-12 02:55:40,425 INFO [Listener at localhost/46667] assignment.AssignmentManager(315): Stopping assignment manager 2023-04-12 02:55:40,425 ERROR [Listener at localhost/46667] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-04-12 02:55:40,425 INFO [Listener at localhost/46667] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-04-12 02:55:40,426 INFO [Listener at localhost/46667] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-04-12 02:55:40,426 INFO [Listener at localhost/46667] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-04-12 02:55:40,426 INFO [Listener at localhost/46667] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-04-12 02:55:40,426 INFO [Listener at localhost/46667] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-04-12 02:55:40,429 DEBUG [Listener at localhost/46667] store.ProcedureTree(114): Procedure Procedure(pid=112, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[0, 1, 2, 7] 2023-04-12 02:55:40,429 DEBUG [Listener at localhost/46667] store.ProcedureTree(114): Procedure Procedure(pid=113, ppid=112, class=org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure) stack ids=[3, 6] 2023-04-12 02:55:40,429 DEBUG [Listener at localhost/46667] store.ProcedureTree(114): Procedure Procedure(pid=114, ppid=113, class=org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure) stack ids=[4, 5] 2023-04-12 02:55:40,429 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-04-12 02:55:40,430 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-04-12 02:55:40,430 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-04-12 02:55:40,430 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-04-12 02:55:40,430 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-12 02:55:40,430 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=cc9012ec0b656baa24d04368fbe026c4, daughterA=e2a9b8fe15a781798c0b49e48ff5aad7, daughterB=6f55b5d6a460bd8c7569dee22f049104 2023-04-12 02:55:40,430 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-12 02:55:40,431 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-12 02:55:40,431 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-04-12 02:55:40,431 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=28, state=SUCCESS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=315e13e868a7f68a3ce37802b2f6312d, daughterA=4e1092f3c00484f8f09c730861225d43, daughterB=291734c00d212fd203b04cf8872d4649 2023-04-12 02:55:40,431 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=35, state=SUCCESS; DisableTableProcedure table=testSplitWithoutPONR 2023-04-12 02:55:40,431 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=40, state=SUCCESS; DeleteTableProcedure table=testSplitWithoutPONR 2023-04-12 02:55:40,431 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=41, state=SUCCESS; CreateTableProcedure table=testRollbackAndDoubleExecution 2023-04-12 02:55:40,432 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=44, state=ROLLEDBACK, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=96cdae2f7638212730b694522b9a464a, daughterA=fdd18e9cf539c13d863267ce16031583, daughterB=578095173fbc35e181cce14464037c2e 2023-04-12 02:55:40,432 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=47, state=SUCCESS; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=96cdae2f7638212730b694522b9a464a, ASSIGN 2023-04-12 02:55:40,432 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=49, state=SUCCESS; DisableTableProcedure table=testRollbackAndDoubleExecution 2023-04-12 02:55:40,432 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=52, state=SUCCESS; DeleteTableProcedure table=testRollbackAndDoubleExecution 2023-04-12 02:55:40,432 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=53, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionNoStoreFile 2023-04-12 02:55:40,432 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=56, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionNoStoreFile, parent=947433d660677a65ad9876ce2025e3e0, daughterA=b24cea6994eaa39f3128573197d02d9b, daughterB=b0e3d5fd7da33cb33084791956f0d7ca 2023-04-12 02:55:40,433 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=63, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionNoStoreFile 2023-04-12 02:55:40,433 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=68, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionNoStoreFile 2023-04-12 02:55:40,433 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=69, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:55:40,433 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=72, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionDeletedRowsDaughter, parent=1c6b8f18205322cbb248ea4fb10155d8, daughterA=81296e35c0d27ffa1ba7676ff53f87e1, daughterB=72b78dfd0678a920ddc3700612ff3932 2023-04-12 02:55:40,433 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=79, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:55:40,434 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=84, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:55:40,434 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=85, state=SUCCESS; CreateTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-04-12 02:55:40,434 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=88, state=SUCCESS; ModifyTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-04-12 02:55:40,435 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=95, state=ROLLEDBACK, exception=org.apache.hadoop.hbase.HBaseIOException via master-split-regions:org.apache.hadoop.hbase.HBaseIOException: The parent region state=OPENING, location=jenkins-hbase4.apache.org,34579,1681268031719, table=testRollbackForSplitTableRegionWithReplica, region=94855f8e18950d1c0c1e1d85df03f9ad is currently in transition, give up; SplitTableRegionProcedure table=testRollbackForSplitTableRegionWithReplica, parent=4381c9c8e57c5dbb2533908abaf07cf1, daughterA=2e35c2f6b8767f438f484c700a605299, daughterB=8407ddb6fc7704d121cd7f8a674aefb6 2023-04-12 02:55:40,435 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=96, state=SUCCESS; DisableTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-04-12 02:55:40,435 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=101, state=SUCCESS; DeleteTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-04-12 02:55:40,435 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=102, state=SUCCESS; CreateTableProcedure table=testInvalidSplitKey 2023-04-12 02:55:40,435 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=105, state=SUCCESS; DisableTableProcedure table=testInvalidSplitKey 2023-04-12 02:55:40,435 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=108, state=SUCCESS; DeleteTableProcedure table=testInvalidSplitKey 2023-04-12 02:55:40,436 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=109, state=SUCCESS; CreateTableProcedure table=testRecoveryAndDoubleExecution 2023-04-12 02:55:40,436 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(468): Loading pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_CREATE_DAUGHTER_REGIONS; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=c09b5f7c44d4a83602090cbccc082b61, daughterA=25b84f3651d3c0adac7c0275b4cb1e0c, daughterB=205d5359b7ecb778c3d784ef1bb72d34 2023-04-12 02:55:40,436 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(468): Loading pid=113, ppid=112, state=SUCCESS; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=c09b5f7c44d4a83602090cbccc082b61, UNASSIGN 2023-04-12 02:55:40,436 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(468): Loading pid=114, ppid=113, state=SUCCESS; CloseRegionProcedure c09b5f7c44d4a83602090cbccc082b61, server=jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:40,436 DEBUG [Listener at localhost/46667] procedure2.Procedure(966): pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_CREATE_DAUGHTER_REGIONS; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=c09b5f7c44d4a83602090cbccc082b61, daughterA=25b84f3651d3c0adac7c0275b4cb1e0c, daughterB=205d5359b7ecb778c3d784ef1bb72d34 held the lock before restarting, call acquireLock to restore it. 2023-04-12 02:55:40,437 INFO [Listener at localhost/46667] procedure.MasterProcedureScheduler(727): Took xlock for pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_CREATE_DAUGHTER_REGIONS; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=c09b5f7c44d4a83602090cbccc082b61, daughterA=25b84f3651d3c0adac7c0275b4cb1e0c, daughterB=205d5359b7ecb778c3d784ef1bb72d34 2023-04-12 02:55:40,437 INFO [Listener at localhost/46667] procedure.MasterProcedureScheduler(727): Took xlock for pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_CREATE_DAUGHTER_REGIONS; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=c09b5f7c44d4a83602090cbccc082b61, daughterA=25b84f3651d3c0adac7c0275b4cb1e0c, daughterB=205d5359b7ecb778c3d784ef1bb72d34 2023-04-12 02:55:40,437 INFO [Listener at localhost/46667] procedure.MasterProcedureScheduler(727): Took xlock for pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_CREATE_DAUGHTER_REGIONS; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=c09b5f7c44d4a83602090cbccc082b61, daughterA=25b84f3651d3c0adac7c0275b4cb1e0c, daughterB=205d5359b7ecb778c3d784ef1bb72d34 2023-04-12 02:55:40,437 DEBUG [Listener at localhost/46667] procedure2.Procedure(946): pid=113, ppid=112, state=SUCCESS; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=c09b5f7c44d4a83602090cbccc082b61, UNASSIGN didn't hold the lock before restarting, skip acquiring lock. 2023-04-12 02:55:40,437 DEBUG [Listener at localhost/46667] procedure2.Procedure(946): pid=114, ppid=113, state=SUCCESS; CloseRegionProcedure c09b5f7c44d4a83602090cbccc082b61, server=jenkins-hbase4.apache.org,43091,1681268031804 didn't hold the lock before restarting, skip acquiring lock. 2023-04-12 02:55:40,437 INFO [Listener at localhost/46667] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 10 msec 2023-04-12 02:55:40,442 DEBUG [Listener at localhost/46667] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-04-12 02:55:40,443 DEBUG [Listener at localhost/46667] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase4.apache.org,44679,1681268031772, table=hbase:meta, region=1588230740 2023-04-12 02:55:40,443 INFO [Listener at localhost/46667] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1681268170443 2023-04-12 02:55:40,445 DEBUG [Listener at localhost/46667] assignment.AssignmentManager(1517): Joining cluster... 2023-04-12 02:55:40,445 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_CREATE_DAUGHTER_REGIONS; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=c09b5f7c44d4a83602090cbccc082b61, daughterA=25b84f3651d3c0adac7c0275b4cb1e0c, daughterB=205d5359b7ecb778c3d784ef1bb72d34 2023-04-12 02:55:40,447 DEBUG [Listener at localhost/46667] assignment.RegionStateStore(147): Load hbase:meta entry region=1f255e980a31bb266f2206a30aa2ae3f, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,44679,1681268031772, regionLocation=jenkins-hbase4.apache.org,44679,1681268031772, openSeqNum=2 2023-04-12 02:55:40,448 DEBUG [Listener at localhost/46667] assignment.RegionStateStore(147): Load hbase:meta entry region=c09b5f7c44d4a83602090cbccc082b61, regionState=CLOSED, lastHost=jenkins-hbase4.apache.org,43091,1681268031804, regionLocation=jenkins-hbase4.apache.org,43091,1681268031804, openSeqNum=2 2023-04-12 02:55:40,448 INFO [Listener at localhost/46667] assignment.AssignmentManager(1529): Number of RegionServers=3 2023-04-12 02:55:40,448 INFO [Listener at localhost/46667] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1681268200448 2023-04-12 02:55:40,448 INFO [Listener at localhost/46667] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1681268260448 2023-04-12 02:55:40,448 INFO [Listener at localhost/46667] assignment.AssignmentManager(1536): Joined the cluster in 3 msec 2023-04-12 02:55:40,458 INFO [PEWorker-1] assignment.SplitTableRegionProcedure(694): pid=112 splitting 14 storefiles, region=c09b5f7c44d4a83602090cbccc082b61, threads=14 2023-04-12 02:55:40,458 DEBUG [StoreFileSplitter-pool-0] assignment.SplitTableRegionProcedure(776): pid=112 splitting started for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf2/1b3fc837d85449cba8b914aa7f18f0fe for region: c09b5f7c44d4a83602090cbccc082b61 2023-04-12 02:55:40,458 DEBUG [StoreFileSplitter-pool-1] assignment.SplitTableRegionProcedure(776): pid=112 splitting started for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf2/5b22fe6156a04c8c9486fd6f189bf4d7 for region: c09b5f7c44d4a83602090cbccc082b61 2023-04-12 02:55:40,460 DEBUG [StoreFileSplitter-pool-2] assignment.SplitTableRegionProcedure(776): pid=112 splitting started for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf2/a4e6a82328b24916838ccb970ddd7704 for region: c09b5f7c44d4a83602090cbccc082b61 2023-04-12 02:55:40,460 DEBUG [StoreFileSplitter-pool-3] assignment.SplitTableRegionProcedure(776): pid=112 splitting started for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf2/a7d4784e93134ef2943643862fd08463 for region: c09b5f7c44d4a83602090cbccc082b61 2023-04-12 02:55:40,464 DEBUG [StoreFileSplitter-pool-4] assignment.SplitTableRegionProcedure(776): pid=112 splitting started for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf2/f70a7ebc6c464a08ab00a25e137bbc72 for region: c09b5f7c44d4a83602090cbccc082b61 2023-04-12 02:55:40,468 DEBUG [StoreFileSplitter-pool-5] assignment.SplitTableRegionProcedure(776): pid=112 splitting started for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf2/fb8ce8dc74bd47fabd120f6f688962d5 for region: c09b5f7c44d4a83602090cbccc082b61 2023-04-12 02:55:40,468 DEBUG [StoreFileSplitter-pool-6] assignment.SplitTableRegionProcedure(776): pid=112 splitting started for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf2/fe343f29642942069fb8e38396f28358 for region: c09b5f7c44d4a83602090cbccc082b61 2023-04-12 02:55:40,469 DEBUG [StoreFileSplitter-pool-7] assignment.SplitTableRegionProcedure(776): pid=112 splitting started for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf1/001e134c60444ac2b388f9df6bf5670b for region: c09b5f7c44d4a83602090cbccc082b61 2023-04-12 02:55:40,473 DEBUG [StoreFileSplitter-pool-9] assignment.SplitTableRegionProcedure(776): pid=112 splitting started for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf1/63025bd542924f2caced3f22c416a5a1 for region: c09b5f7c44d4a83602090cbccc082b61 2023-04-12 02:55:40,473 DEBUG [StoreFileSplitter-pool-8] assignment.SplitTableRegionProcedure(776): pid=112 splitting started for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf1/1373dcc8adeb4ae4b9929ef46fbb4980 for region: c09b5f7c44d4a83602090cbccc082b61 2023-04-12 02:55:40,484 DEBUG [StoreFileSplitter-pool-10] assignment.SplitTableRegionProcedure(776): pid=112 splitting started for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf1/9d5ec6ededfd4de8b6d44e635d18346c for region: c09b5f7c44d4a83602090cbccc082b61 2023-04-12 02:55:40,514 DEBUG [StoreFileSplitter-pool-11] assignment.SplitTableRegionProcedure(776): pid=112 splitting started for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf1/a79c7cc5c59c4f6bbe6a6a584d824f36 for region: c09b5f7c44d4a83602090cbccc082b61 2023-04-12 02:55:40,515 DEBUG [StoreFileSplitter-pool-12] assignment.SplitTableRegionProcedure(776): pid=112 splitting started for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf1/c14bd2fe66004d6d842c72f4b2a4ee90 for region: c09b5f7c44d4a83602090cbccc082b61 2023-04-12 02:55:40,515 DEBUG [StoreFileSplitter-pool-13] assignment.SplitTableRegionProcedure(776): pid=112 splitting started for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf1/e62dda4fe11148e88206bdd397ca64c0 for region: c09b5f7c44d4a83602090cbccc082b61 2023-04-12 02:55:40,805 DEBUG [StoreFileSplitter-pool-6] assignment.SplitTableRegionProcedure(787): pid=112 splitting complete for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf2/fe343f29642942069fb8e38396f28358 for region: c09b5f7c44d4a83602090cbccc082b61 2023-04-12 02:55:40,808 DEBUG [StoreFileSplitter-pool-2] assignment.SplitTableRegionProcedure(787): pid=112 splitting complete for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf2/a4e6a82328b24916838ccb970ddd7704 for region: c09b5f7c44d4a83602090cbccc082b61 2023-04-12 02:55:40,828 DEBUG [StoreFileSplitter-pool-5] assignment.SplitTableRegionProcedure(787): pid=112 splitting complete for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf2/fb8ce8dc74bd47fabd120f6f688962d5 for region: c09b5f7c44d4a83602090cbccc082b61 2023-04-12 02:55:40,829 DEBUG [StoreFileSplitter-pool-8] assignment.SplitTableRegionProcedure(787): pid=112 splitting complete for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf1/1373dcc8adeb4ae4b9929ef46fbb4980 for region: c09b5f7c44d4a83602090cbccc082b61 2023-04-12 02:55:40,829 DEBUG [StoreFileSplitter-pool-0] assignment.SplitTableRegionProcedure(787): pid=112 splitting complete for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf2/1b3fc837d85449cba8b914aa7f18f0fe for region: c09b5f7c44d4a83602090cbccc082b61 2023-04-12 02:55:40,844 DEBUG [StoreFileSplitter-pool-4] assignment.SplitTableRegionProcedure(787): pid=112 splitting complete for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf2/f70a7ebc6c464a08ab00a25e137bbc72 for region: c09b5f7c44d4a83602090cbccc082b61 2023-04-12 02:55:40,846 DEBUG [StoreFileSplitter-pool-3] assignment.SplitTableRegionProcedure(787): pid=112 splitting complete for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf2/a7d4784e93134ef2943643862fd08463 for region: c09b5f7c44d4a83602090cbccc082b61 2023-04-12 02:55:40,855 DEBUG [StoreFileSplitter-pool-7] assignment.SplitTableRegionProcedure(787): pid=112 splitting complete for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf1/001e134c60444ac2b388f9df6bf5670b for region: c09b5f7c44d4a83602090cbccc082b61 2023-04-12 02:55:40,860 DEBUG [StoreFileSplitter-pool-12] assignment.SplitTableRegionProcedure(787): pid=112 splitting complete for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf1/c14bd2fe66004d6d842c72f4b2a4ee90 for region: c09b5f7c44d4a83602090cbccc082b61 2023-04-12 02:55:40,860 DEBUG [StoreFileSplitter-pool-11] assignment.SplitTableRegionProcedure(787): pid=112 splitting complete for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf1/a79c7cc5c59c4f6bbe6a6a584d824f36 for region: c09b5f7c44d4a83602090cbccc082b61 2023-04-12 02:55:40,876 DEBUG [StoreFileSplitter-pool-13] assignment.SplitTableRegionProcedure(787): pid=112 splitting complete for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf1/e62dda4fe11148e88206bdd397ca64c0 for region: c09b5f7c44d4a83602090cbccc082b61 2023-04-12 02:55:40,876 DEBUG [StoreFileSplitter-pool-10] assignment.SplitTableRegionProcedure(787): pid=112 splitting complete for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf1/9d5ec6ededfd4de8b6d44e635d18346c for region: c09b5f7c44d4a83602090cbccc082b61 2023-04-12 02:55:40,888 DEBUG [StoreFileSplitter-pool-9] assignment.SplitTableRegionProcedure(787): pid=112 splitting complete for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf1/63025bd542924f2caced3f22c416a5a1 for region: c09b5f7c44d4a83602090cbccc082b61 2023-04-12 02:55:40,907 DEBUG [StoreFileSplitter-pool-1] assignment.SplitTableRegionProcedure(787): pid=112 splitting complete for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf2/5b22fe6156a04c8c9486fd6f189bf4d7 for region: c09b5f7c44d4a83602090cbccc082b61 2023-04-12 02:55:40,907 DEBUG [PEWorker-1] assignment.SplitTableRegionProcedure(755): pid=112 split storefiles for region c09b5f7c44d4a83602090cbccc082b61 Daughter A: 14 storefiles, Daughter B: 14 storefiles. 2023-04-12 02:55:40,955 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-04-12 02:55:40,960 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/25b84f3651d3c0adac7c0275b4cb1e0c/recovered.edits/85.seqid, newMaxSeqId=85, maxSeqId=-1 2023-04-12 02:55:40,962 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/205d5359b7ecb778c3d784ef1bb72d34/recovered.edits/85.seqid, newMaxSeqId=85, maxSeqId=-1 2023-04-12 02:55:40,962 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-04-12 02:55:40,962 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_BEFORE_META, locked=true; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=c09b5f7c44d4a83602090cbccc082b61, daughterA=25b84f3651d3c0adac7c0275b4cb1e0c, daughterB=205d5359b7ecb778c3d784ef1bb72d34 2023-04-12 02:55:40,962 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-04-12 02:55:40,962 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_BEFORE_META, locked=true; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=c09b5f7c44d4a83602090cbccc082b61, daughterA=25b84f3651d3c0adac7c0275b4cb1e0c, daughterB=205d5359b7ecb778c3d784ef1bb72d34 at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-04-12 02:55:41,199 INFO [Listener at localhost/46667] procedure.MasterProcedureTestingUtility(454): Restart 5 exec state=pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_BEFORE_META, locked=true; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=c09b5f7c44d4a83602090cbccc082b61, daughterA=25b84f3651d3c0adac7c0275b4cb1e0c, daughterB=205d5359b7ecb778c3d784ef1bb72d34 2023-04-12 02:55:41,199 INFO [Listener at localhost/46667] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-04-12 02:55:41,199 INFO [Listener at localhost/46667] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-04-12 02:55:41,199 INFO [Listener at localhost/46667] assignment.AssignmentManager(315): Stopping assignment manager 2023-04-12 02:55:41,199 ERROR [Listener at localhost/46667] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-04-12 02:55:41,199 INFO [Listener at localhost/46667] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-04-12 02:55:41,199 INFO [Listener at localhost/46667] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-04-12 02:55:41,199 INFO [Listener at localhost/46667] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-04-12 02:55:41,199 INFO [Listener at localhost/46667] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-04-12 02:55:41,200 INFO [Listener at localhost/46667] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-04-12 02:55:41,201 DEBUG [Listener at localhost/46667] store.ProcedureTree(114): Procedure Procedure(pid=112, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[0, 1, 2, 7, 8] 2023-04-12 02:55:41,201 DEBUG [Listener at localhost/46667] store.ProcedureTree(114): Procedure Procedure(pid=113, ppid=112, class=org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure) stack ids=[3, 6] 2023-04-12 02:55:41,201 DEBUG [Listener at localhost/46667] store.ProcedureTree(114): Procedure Procedure(pid=114, ppid=113, class=org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure) stack ids=[4, 5] 2023-04-12 02:55:41,201 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-04-12 02:55:41,202 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-04-12 02:55:41,202 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-04-12 02:55:41,202 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-04-12 02:55:41,202 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-12 02:55:41,203 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=cc9012ec0b656baa24d04368fbe026c4, daughterA=e2a9b8fe15a781798c0b49e48ff5aad7, daughterB=6f55b5d6a460bd8c7569dee22f049104 2023-04-12 02:55:41,203 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-12 02:55:41,203 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-12 02:55:41,203 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-04-12 02:55:41,203 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=28, state=SUCCESS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=315e13e868a7f68a3ce37802b2f6312d, daughterA=4e1092f3c00484f8f09c730861225d43, daughterB=291734c00d212fd203b04cf8872d4649 2023-04-12 02:55:41,203 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=35, state=SUCCESS; DisableTableProcedure table=testSplitWithoutPONR 2023-04-12 02:55:41,203 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=40, state=SUCCESS; DeleteTableProcedure table=testSplitWithoutPONR 2023-04-12 02:55:41,204 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=41, state=SUCCESS; CreateTableProcedure table=testRollbackAndDoubleExecution 2023-04-12 02:55:41,204 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=44, state=ROLLEDBACK, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=96cdae2f7638212730b694522b9a464a, daughterA=fdd18e9cf539c13d863267ce16031583, daughterB=578095173fbc35e181cce14464037c2e 2023-04-12 02:55:41,204 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=47, state=SUCCESS; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=96cdae2f7638212730b694522b9a464a, ASSIGN 2023-04-12 02:55:41,204 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=49, state=SUCCESS; DisableTableProcedure table=testRollbackAndDoubleExecution 2023-04-12 02:55:41,204 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=52, state=SUCCESS; DeleteTableProcedure table=testRollbackAndDoubleExecution 2023-04-12 02:55:41,204 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=53, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionNoStoreFile 2023-04-12 02:55:41,205 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=56, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionNoStoreFile, parent=947433d660677a65ad9876ce2025e3e0, daughterA=b24cea6994eaa39f3128573197d02d9b, daughterB=b0e3d5fd7da33cb33084791956f0d7ca 2023-04-12 02:55:41,205 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=63, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionNoStoreFile 2023-04-12 02:55:41,205 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=68, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionNoStoreFile 2023-04-12 02:55:41,205 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=69, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:55:41,205 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=72, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionDeletedRowsDaughter, parent=1c6b8f18205322cbb248ea4fb10155d8, daughterA=81296e35c0d27ffa1ba7676ff53f87e1, daughterB=72b78dfd0678a920ddc3700612ff3932 2023-04-12 02:55:41,205 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=79, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:55:41,205 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=84, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:55:41,205 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=85, state=SUCCESS; CreateTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-04-12 02:55:41,206 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=88, state=SUCCESS; ModifyTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-04-12 02:55:41,206 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=95, state=ROLLEDBACK, exception=org.apache.hadoop.hbase.HBaseIOException via master-split-regions:org.apache.hadoop.hbase.HBaseIOException: The parent region state=OPENING, location=jenkins-hbase4.apache.org,34579,1681268031719, table=testRollbackForSplitTableRegionWithReplica, region=94855f8e18950d1c0c1e1d85df03f9ad is currently in transition, give up; SplitTableRegionProcedure table=testRollbackForSplitTableRegionWithReplica, parent=4381c9c8e57c5dbb2533908abaf07cf1, daughterA=2e35c2f6b8767f438f484c700a605299, daughterB=8407ddb6fc7704d121cd7f8a674aefb6 2023-04-12 02:55:41,206 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=96, state=SUCCESS; DisableTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-04-12 02:55:41,206 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=101, state=SUCCESS; DeleteTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-04-12 02:55:41,206 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=102, state=SUCCESS; CreateTableProcedure table=testInvalidSplitKey 2023-04-12 02:55:41,206 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=105, state=SUCCESS; DisableTableProcedure table=testInvalidSplitKey 2023-04-12 02:55:41,206 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=108, state=SUCCESS; DeleteTableProcedure table=testInvalidSplitKey 2023-04-12 02:55:41,207 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=109, state=SUCCESS; CreateTableProcedure table=testRecoveryAndDoubleExecution 2023-04-12 02:55:41,207 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(468): Loading pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_WRITE_MAX_SEQUENCE_ID_FILE; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=c09b5f7c44d4a83602090cbccc082b61, daughterA=25b84f3651d3c0adac7c0275b4cb1e0c, daughterB=205d5359b7ecb778c3d784ef1bb72d34 2023-04-12 02:55:41,207 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(468): Loading pid=113, ppid=112, state=SUCCESS; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=c09b5f7c44d4a83602090cbccc082b61, UNASSIGN 2023-04-12 02:55:41,207 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(468): Loading pid=114, ppid=113, state=SUCCESS; CloseRegionProcedure c09b5f7c44d4a83602090cbccc082b61, server=jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:41,207 DEBUG [Listener at localhost/46667] procedure2.Procedure(966): pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_WRITE_MAX_SEQUENCE_ID_FILE; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=c09b5f7c44d4a83602090cbccc082b61, daughterA=25b84f3651d3c0adac7c0275b4cb1e0c, daughterB=205d5359b7ecb778c3d784ef1bb72d34 held the lock before restarting, call acquireLock to restore it. 2023-04-12 02:55:41,207 INFO [Listener at localhost/46667] procedure.MasterProcedureScheduler(727): Took xlock for pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_WRITE_MAX_SEQUENCE_ID_FILE; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=c09b5f7c44d4a83602090cbccc082b61, daughterA=25b84f3651d3c0adac7c0275b4cb1e0c, daughterB=205d5359b7ecb778c3d784ef1bb72d34 2023-04-12 02:55:41,207 INFO [Listener at localhost/46667] procedure.MasterProcedureScheduler(727): Took xlock for pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_WRITE_MAX_SEQUENCE_ID_FILE; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=c09b5f7c44d4a83602090cbccc082b61, daughterA=25b84f3651d3c0adac7c0275b4cb1e0c, daughterB=205d5359b7ecb778c3d784ef1bb72d34 2023-04-12 02:55:41,207 INFO [Listener at localhost/46667] procedure.MasterProcedureScheduler(727): Took xlock for pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_WRITE_MAX_SEQUENCE_ID_FILE; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=c09b5f7c44d4a83602090cbccc082b61, daughterA=25b84f3651d3c0adac7c0275b4cb1e0c, daughterB=205d5359b7ecb778c3d784ef1bb72d34 2023-04-12 02:55:41,207 DEBUG [Listener at localhost/46667] procedure2.Procedure(946): pid=113, ppid=112, state=SUCCESS; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=c09b5f7c44d4a83602090cbccc082b61, UNASSIGN didn't hold the lock before restarting, skip acquiring lock. 2023-04-12 02:55:41,208 DEBUG [Listener at localhost/46667] procedure2.Procedure(946): pid=114, ppid=113, state=SUCCESS; CloseRegionProcedure c09b5f7c44d4a83602090cbccc082b61, server=jenkins-hbase4.apache.org,43091,1681268031804 didn't hold the lock before restarting, skip acquiring lock. 2023-04-12 02:55:41,208 INFO [Listener at localhost/46667] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 7 msec 2023-04-12 02:55:41,208 DEBUG [Listener at localhost/46667] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-04-12 02:55:41,209 DEBUG [Listener at localhost/46667] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase4.apache.org,44679,1681268031772, table=hbase:meta, region=1588230740 2023-04-12 02:55:41,209 INFO [Listener at localhost/46667] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1681268171209 2023-04-12 02:55:41,211 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_WRITE_MAX_SEQUENCE_ID_FILE; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=c09b5f7c44d4a83602090cbccc082b61, daughterA=25b84f3651d3c0adac7c0275b4cb1e0c, daughterB=205d5359b7ecb778c3d784ef1bb72d34 2023-04-12 02:55:41,211 DEBUG [Listener at localhost/46667] assignment.AssignmentManager(1517): Joining cluster... 2023-04-12 02:55:41,214 DEBUG [Listener at localhost/46667] assignment.RegionStateStore(147): Load hbase:meta entry region=1f255e980a31bb266f2206a30aa2ae3f, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,44679,1681268031772, regionLocation=jenkins-hbase4.apache.org,44679,1681268031772, openSeqNum=2 2023-04-12 02:55:41,214 DEBUG [Listener at localhost/46667] assignment.RegionStateStore(147): Load hbase:meta entry region=c09b5f7c44d4a83602090cbccc082b61, regionState=CLOSED, lastHost=jenkins-hbase4.apache.org,43091,1681268031804, regionLocation=jenkins-hbase4.apache.org,43091,1681268031804, openSeqNum=2 2023-04-12 02:55:41,214 INFO [Listener at localhost/46667] assignment.AssignmentManager(1529): Number of RegionServers=3 2023-04-12 02:55:41,215 INFO [Listener at localhost/46667] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1681268201215 2023-04-12 02:55:41,215 INFO [Listener at localhost/46667] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1681268261215 2023-04-12 02:55:41,215 INFO [Listener at localhost/46667] assignment.AssignmentManager(1536): Joined the cluster in 3 msec 2023-04-12 02:55:41,217 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-04-12 02:55:41,219 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-04-12 02:55:41,219 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_UPDATE_META, locked=true; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=c09b5f7c44d4a83602090cbccc082b61, daughterA=25b84f3651d3c0adac7c0275b4cb1e0c, daughterB=205d5359b7ecb778c3d784ef1bb72d34 2023-04-12 02:55:41,219 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-04-12 02:55:41,219 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_UPDATE_META, locked=true; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=c09b5f7c44d4a83602090cbccc082b61, daughterA=25b84f3651d3c0adac7c0275b4cb1e0c, daughterB=205d5359b7ecb778c3d784ef1bb72d34 at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-04-12 02:55:41,465 INFO [Listener at localhost/46667] procedure.MasterProcedureTestingUtility(454): Restart 6 exec state=pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_UPDATE_META, locked=true; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=c09b5f7c44d4a83602090cbccc082b61, daughterA=25b84f3651d3c0adac7c0275b4cb1e0c, daughterB=205d5359b7ecb778c3d784ef1bb72d34 2023-04-12 02:55:41,465 INFO [Listener at localhost/46667] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-04-12 02:55:41,465 INFO [Listener at localhost/46667] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-04-12 02:55:41,465 INFO [Listener at localhost/46667] assignment.AssignmentManager(315): Stopping assignment manager 2023-04-12 02:55:41,465 ERROR [Listener at localhost/46667] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-04-12 02:55:41,466 INFO [Listener at localhost/46667] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-04-12 02:55:41,466 INFO [Listener at localhost/46667] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-04-12 02:55:41,466 INFO [Listener at localhost/46667] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-04-12 02:55:41,466 INFO [Listener at localhost/46667] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-04-12 02:55:41,466 INFO [Listener at localhost/46667] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-04-12 02:55:41,467 DEBUG [Listener at localhost/46667] store.ProcedureTree(114): Procedure Procedure(pid=112, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[0, 1, 2, 7, 8, 9] 2023-04-12 02:55:41,468 DEBUG [Listener at localhost/46667] store.ProcedureTree(114): Procedure Procedure(pid=113, ppid=112, class=org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure) stack ids=[3, 6] 2023-04-12 02:55:41,468 DEBUG [Listener at localhost/46667] store.ProcedureTree(114): Procedure Procedure(pid=114, ppid=113, class=org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure) stack ids=[4, 5] 2023-04-12 02:55:41,468 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-04-12 02:55:41,468 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-04-12 02:55:41,468 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-04-12 02:55:41,468 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-04-12 02:55:41,469 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-12 02:55:41,469 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=cc9012ec0b656baa24d04368fbe026c4, daughterA=e2a9b8fe15a781798c0b49e48ff5aad7, daughterB=6f55b5d6a460bd8c7569dee22f049104 2023-04-12 02:55:41,469 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-12 02:55:41,469 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-12 02:55:41,469 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-04-12 02:55:41,470 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=28, state=SUCCESS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=315e13e868a7f68a3ce37802b2f6312d, daughterA=4e1092f3c00484f8f09c730861225d43, daughterB=291734c00d212fd203b04cf8872d4649 2023-04-12 02:55:41,470 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=35, state=SUCCESS; DisableTableProcedure table=testSplitWithoutPONR 2023-04-12 02:55:41,470 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=40, state=SUCCESS; DeleteTableProcedure table=testSplitWithoutPONR 2023-04-12 02:55:41,470 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=41, state=SUCCESS; CreateTableProcedure table=testRollbackAndDoubleExecution 2023-04-12 02:55:41,470 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=44, state=ROLLEDBACK, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=96cdae2f7638212730b694522b9a464a, daughterA=fdd18e9cf539c13d863267ce16031583, daughterB=578095173fbc35e181cce14464037c2e 2023-04-12 02:55:41,470 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=47, state=SUCCESS; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=96cdae2f7638212730b694522b9a464a, ASSIGN 2023-04-12 02:55:41,470 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=49, state=SUCCESS; DisableTableProcedure table=testRollbackAndDoubleExecution 2023-04-12 02:55:41,471 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=52, state=SUCCESS; DeleteTableProcedure table=testRollbackAndDoubleExecution 2023-04-12 02:55:41,471 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=53, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionNoStoreFile 2023-04-12 02:55:41,471 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=56, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionNoStoreFile, parent=947433d660677a65ad9876ce2025e3e0, daughterA=b24cea6994eaa39f3128573197d02d9b, daughterB=b0e3d5fd7da33cb33084791956f0d7ca 2023-04-12 02:55:41,471 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=63, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionNoStoreFile 2023-04-12 02:55:41,471 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=68, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionNoStoreFile 2023-04-12 02:55:41,471 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=69, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:55:41,471 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=72, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionDeletedRowsDaughter, parent=1c6b8f18205322cbb248ea4fb10155d8, daughterA=81296e35c0d27ffa1ba7676ff53f87e1, daughterB=72b78dfd0678a920ddc3700612ff3932 2023-04-12 02:55:41,471 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=79, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:55:41,472 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=84, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:55:41,472 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=85, state=SUCCESS; CreateTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-04-12 02:55:41,472 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=88, state=SUCCESS; ModifyTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-04-12 02:55:41,472 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=95, state=ROLLEDBACK, exception=org.apache.hadoop.hbase.HBaseIOException via master-split-regions:org.apache.hadoop.hbase.HBaseIOException: The parent region state=OPENING, location=jenkins-hbase4.apache.org,34579,1681268031719, table=testRollbackForSplitTableRegionWithReplica, region=94855f8e18950d1c0c1e1d85df03f9ad is currently in transition, give up; SplitTableRegionProcedure table=testRollbackForSplitTableRegionWithReplica, parent=4381c9c8e57c5dbb2533908abaf07cf1, daughterA=2e35c2f6b8767f438f484c700a605299, daughterB=8407ddb6fc7704d121cd7f8a674aefb6 2023-04-12 02:55:41,472 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=96, state=SUCCESS; DisableTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-04-12 02:55:41,473 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=101, state=SUCCESS; DeleteTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-04-12 02:55:41,473 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=102, state=SUCCESS; CreateTableProcedure table=testInvalidSplitKey 2023-04-12 02:55:41,473 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=105, state=SUCCESS; DisableTableProcedure table=testInvalidSplitKey 2023-04-12 02:55:41,473 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=108, state=SUCCESS; DeleteTableProcedure table=testInvalidSplitKey 2023-04-12 02:55:41,473 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=109, state=SUCCESS; CreateTableProcedure table=testRecoveryAndDoubleExecution 2023-04-12 02:55:41,474 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(468): Loading pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_BEFORE_META; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=c09b5f7c44d4a83602090cbccc082b61, daughterA=25b84f3651d3c0adac7c0275b4cb1e0c, daughterB=205d5359b7ecb778c3d784ef1bb72d34 2023-04-12 02:55:41,474 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(468): Loading pid=113, ppid=112, state=SUCCESS; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=c09b5f7c44d4a83602090cbccc082b61, UNASSIGN 2023-04-12 02:55:41,474 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(468): Loading pid=114, ppid=113, state=SUCCESS; CloseRegionProcedure c09b5f7c44d4a83602090cbccc082b61, server=jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:41,474 DEBUG [Listener at localhost/46667] procedure2.Procedure(966): pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_BEFORE_META; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=c09b5f7c44d4a83602090cbccc082b61, daughterA=25b84f3651d3c0adac7c0275b4cb1e0c, daughterB=205d5359b7ecb778c3d784ef1bb72d34 held the lock before restarting, call acquireLock to restore it. 2023-04-12 02:55:41,474 INFO [Listener at localhost/46667] procedure.MasterProcedureScheduler(727): Took xlock for pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_BEFORE_META; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=c09b5f7c44d4a83602090cbccc082b61, daughterA=25b84f3651d3c0adac7c0275b4cb1e0c, daughterB=205d5359b7ecb778c3d784ef1bb72d34 2023-04-12 02:55:41,474 INFO [Listener at localhost/46667] procedure.MasterProcedureScheduler(727): Took xlock for pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_BEFORE_META; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=c09b5f7c44d4a83602090cbccc082b61, daughterA=25b84f3651d3c0adac7c0275b4cb1e0c, daughterB=205d5359b7ecb778c3d784ef1bb72d34 2023-04-12 02:55:41,474 INFO [Listener at localhost/46667] procedure.MasterProcedureScheduler(727): Took xlock for pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_BEFORE_META; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=c09b5f7c44d4a83602090cbccc082b61, daughterA=25b84f3651d3c0adac7c0275b4cb1e0c, daughterB=205d5359b7ecb778c3d784ef1bb72d34 2023-04-12 02:55:41,474 DEBUG [Listener at localhost/46667] procedure2.Procedure(946): pid=113, ppid=112, state=SUCCESS; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=c09b5f7c44d4a83602090cbccc082b61, UNASSIGN didn't hold the lock before restarting, skip acquiring lock. 2023-04-12 02:55:41,474 DEBUG [Listener at localhost/46667] procedure2.Procedure(946): pid=114, ppid=113, state=SUCCESS; CloseRegionProcedure c09b5f7c44d4a83602090cbccc082b61, server=jenkins-hbase4.apache.org,43091,1681268031804 didn't hold the lock before restarting, skip acquiring lock. 2023-04-12 02:55:41,474 INFO [Listener at localhost/46667] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 7 msec 2023-04-12 02:55:41,475 DEBUG [Listener at localhost/46667] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-04-12 02:55:41,475 DEBUG [Listener at localhost/46667] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase4.apache.org,44679,1681268031772, table=hbase:meta, region=1588230740 2023-04-12 02:55:41,476 INFO [Listener at localhost/46667] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1681268171476 2023-04-12 02:55:41,478 DEBUG [Listener at localhost/46667] assignment.AssignmentManager(1517): Joining cluster... 2023-04-12 02:55:41,479 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_BEFORE_META; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=c09b5f7c44d4a83602090cbccc082b61, daughterA=25b84f3651d3c0adac7c0275b4cb1e0c, daughterB=205d5359b7ecb778c3d784ef1bb72d34 2023-04-12 02:55:41,481 DEBUG [Listener at localhost/46667] assignment.RegionStateStore(147): Load hbase:meta entry region=1f255e980a31bb266f2206a30aa2ae3f, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,44679,1681268031772, regionLocation=jenkins-hbase4.apache.org,44679,1681268031772, openSeqNum=2 2023-04-12 02:55:41,481 DEBUG [Listener at localhost/46667] assignment.RegionStateStore(147): Load hbase:meta entry region=c09b5f7c44d4a83602090cbccc082b61, regionState=CLOSED, lastHost=jenkins-hbase4.apache.org,43091,1681268031804, regionLocation=jenkins-hbase4.apache.org,43091,1681268031804, openSeqNum=2 2023-04-12 02:55:41,482 INFO [Listener at localhost/46667] assignment.AssignmentManager(1529): Number of RegionServers=3 2023-04-12 02:55:41,482 INFO [Listener at localhost/46667] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1681268201482 2023-04-12 02:55:41,482 INFO [Listener at localhost/46667] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1681268261482 2023-04-12 02:55:41,482 INFO [Listener at localhost/46667] assignment.AssignmentManager(1536): Joined the cluster in 3 msec 2023-04-12 02:55:41,485 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-04-12 02:55:41,488 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":3,"row":"testRecoveryAndDoubleExecution,,1681268136320.c09b5f7c44d4a83602090cbccc082b61.","families":{"info":[{"qualifier":"regioninfo","vlen":64,"tag":[],"timestamp":"1681268141488"},{"qualifier":"splitA","vlen":66,"tag":[],"timestamp":"1681268141488"},{"qualifier":"splitB","vlen":66,"tag":[],"timestamp":"1681268141488"}]},"ts":"1681268141488"} 2023-04-12 02:55:41,488 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":3,"row":"testRecoveryAndDoubleExecution,,1681268138347.25b84f3651d3c0adac7c0275b4cb1e0c.","families":{"info":[{"qualifier":"regioninfo","vlen":66,"tag":[],"timestamp":"1681268141488"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681268141488"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681268141488"}]},"ts":"1681268141488"} 2023-04-12 02:55:41,488 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":3,"row":"testRecoveryAndDoubleExecution,41,1681268138347.205d5359b7ecb778c3d784ef1bb72d34.","families":{"info":[{"qualifier":"regioninfo","vlen":66,"tag":[],"timestamp":"1681268141488"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681268141488"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681268141488"}]},"ts":"1681268141488"} 2023-04-12 02:55:41,490 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-04-12 02:55:41,490 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_AFTER_META, locked=true; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=c09b5f7c44d4a83602090cbccc082b61, daughterA=25b84f3651d3c0adac7c0275b4cb1e0c, daughterB=205d5359b7ecb778c3d784ef1bb72d34 2023-04-12 02:55:41,490 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-04-12 02:55:41,490 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_AFTER_META, locked=true; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=c09b5f7c44d4a83602090cbccc082b61, daughterA=25b84f3651d3c0adac7c0275b4cb1e0c, daughterB=205d5359b7ecb778c3d784ef1bb72d34 at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-04-12 02:55:41,732 INFO [Listener at localhost/46667] procedure.MasterProcedureTestingUtility(454): Restart 7 exec state=pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_AFTER_META, locked=true; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=c09b5f7c44d4a83602090cbccc082b61, daughterA=25b84f3651d3c0adac7c0275b4cb1e0c, daughterB=205d5359b7ecb778c3d784ef1bb72d34 2023-04-12 02:55:41,732 INFO [Listener at localhost/46667] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-04-12 02:55:41,732 INFO [Listener at localhost/46667] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-04-12 02:55:41,732 INFO [Listener at localhost/46667] assignment.AssignmentManager(315): Stopping assignment manager 2023-04-12 02:55:41,733 ERROR [Listener at localhost/46667] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-04-12 02:55:41,733 INFO [Listener at localhost/46667] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-04-12 02:55:41,733 INFO [Listener at localhost/46667] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-04-12 02:55:41,733 INFO [Listener at localhost/46667] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-04-12 02:55:41,733 INFO [Listener at localhost/46667] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-04-12 02:55:41,733 INFO [Listener at localhost/46667] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-04-12 02:55:41,734 DEBUG [Listener at localhost/46667] store.ProcedureTree(114): Procedure Procedure(pid=112, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[0, 1, 2, 7, 8, 9, 10] 2023-04-12 02:55:41,735 DEBUG [Listener at localhost/46667] store.ProcedureTree(114): Procedure Procedure(pid=113, ppid=112, class=org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure) stack ids=[3, 6] 2023-04-12 02:55:41,735 DEBUG [Listener at localhost/46667] store.ProcedureTree(114): Procedure Procedure(pid=114, ppid=113, class=org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure) stack ids=[4, 5] 2023-04-12 02:55:41,735 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-04-12 02:55:41,735 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-04-12 02:55:41,735 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-04-12 02:55:41,735 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-04-12 02:55:41,736 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-12 02:55:41,736 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=cc9012ec0b656baa24d04368fbe026c4, daughterA=e2a9b8fe15a781798c0b49e48ff5aad7, daughterB=6f55b5d6a460bd8c7569dee22f049104 2023-04-12 02:55:41,736 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-12 02:55:41,736 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-12 02:55:41,737 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-04-12 02:55:41,737 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=28, state=SUCCESS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=315e13e868a7f68a3ce37802b2f6312d, daughterA=4e1092f3c00484f8f09c730861225d43, daughterB=291734c00d212fd203b04cf8872d4649 2023-04-12 02:55:41,737 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=35, state=SUCCESS; DisableTableProcedure table=testSplitWithoutPONR 2023-04-12 02:55:41,737 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=40, state=SUCCESS; DeleteTableProcedure table=testSplitWithoutPONR 2023-04-12 02:55:41,738 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=41, state=SUCCESS; CreateTableProcedure table=testRollbackAndDoubleExecution 2023-04-12 02:55:41,738 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=44, state=ROLLEDBACK, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=96cdae2f7638212730b694522b9a464a, daughterA=fdd18e9cf539c13d863267ce16031583, daughterB=578095173fbc35e181cce14464037c2e 2023-04-12 02:55:41,738 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=47, state=SUCCESS; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=96cdae2f7638212730b694522b9a464a, ASSIGN 2023-04-12 02:55:41,739 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=49, state=SUCCESS; DisableTableProcedure table=testRollbackAndDoubleExecution 2023-04-12 02:55:41,739 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=52, state=SUCCESS; DeleteTableProcedure table=testRollbackAndDoubleExecution 2023-04-12 02:55:41,739 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=53, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionNoStoreFile 2023-04-12 02:55:41,739 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=56, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionNoStoreFile, parent=947433d660677a65ad9876ce2025e3e0, daughterA=b24cea6994eaa39f3128573197d02d9b, daughterB=b0e3d5fd7da33cb33084791956f0d7ca 2023-04-12 02:55:41,739 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=63, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionNoStoreFile 2023-04-12 02:55:41,740 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=68, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionNoStoreFile 2023-04-12 02:55:41,741 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=69, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:55:41,741 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=72, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionDeletedRowsDaughter, parent=1c6b8f18205322cbb248ea4fb10155d8, daughterA=81296e35c0d27ffa1ba7676ff53f87e1, daughterB=72b78dfd0678a920ddc3700612ff3932 2023-04-12 02:55:41,741 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=79, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:55:41,741 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=84, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:55:41,742 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=85, state=SUCCESS; CreateTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-04-12 02:55:41,742 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=88, state=SUCCESS; ModifyTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-04-12 02:55:41,742 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=95, state=ROLLEDBACK, exception=org.apache.hadoop.hbase.HBaseIOException via master-split-regions:org.apache.hadoop.hbase.HBaseIOException: The parent region state=OPENING, location=jenkins-hbase4.apache.org,34579,1681268031719, table=testRollbackForSplitTableRegionWithReplica, region=94855f8e18950d1c0c1e1d85df03f9ad is currently in transition, give up; SplitTableRegionProcedure table=testRollbackForSplitTableRegionWithReplica, parent=4381c9c8e57c5dbb2533908abaf07cf1, daughterA=2e35c2f6b8767f438f484c700a605299, daughterB=8407ddb6fc7704d121cd7f8a674aefb6 2023-04-12 02:55:41,743 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=96, state=SUCCESS; DisableTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-04-12 02:55:41,743 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=101, state=SUCCESS; DeleteTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-04-12 02:55:41,743 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=102, state=SUCCESS; CreateTableProcedure table=testInvalidSplitKey 2023-04-12 02:55:41,743 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=105, state=SUCCESS; DisableTableProcedure table=testInvalidSplitKey 2023-04-12 02:55:41,744 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=108, state=SUCCESS; DeleteTableProcedure table=testInvalidSplitKey 2023-04-12 02:55:41,744 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=109, state=SUCCESS; CreateTableProcedure table=testRecoveryAndDoubleExecution 2023-04-12 02:55:41,744 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(468): Loading pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_UPDATE_META; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=c09b5f7c44d4a83602090cbccc082b61, daughterA=25b84f3651d3c0adac7c0275b4cb1e0c, daughterB=205d5359b7ecb778c3d784ef1bb72d34 2023-04-12 02:55:41,745 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(468): Loading pid=113, ppid=112, state=SUCCESS; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=c09b5f7c44d4a83602090cbccc082b61, UNASSIGN 2023-04-12 02:55:41,746 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(468): Loading pid=114, ppid=113, state=SUCCESS; CloseRegionProcedure c09b5f7c44d4a83602090cbccc082b61, server=jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:41,746 DEBUG [Listener at localhost/46667] procedure2.Procedure(966): pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_UPDATE_META; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=c09b5f7c44d4a83602090cbccc082b61, daughterA=25b84f3651d3c0adac7c0275b4cb1e0c, daughterB=205d5359b7ecb778c3d784ef1bb72d34 held the lock before restarting, call acquireLock to restore it. 2023-04-12 02:55:41,746 INFO [Listener at localhost/46667] procedure.MasterProcedureScheduler(727): Took xlock for pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_UPDATE_META; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=c09b5f7c44d4a83602090cbccc082b61, daughterA=25b84f3651d3c0adac7c0275b4cb1e0c, daughterB=205d5359b7ecb778c3d784ef1bb72d34 2023-04-12 02:55:41,746 INFO [Listener at localhost/46667] procedure.MasterProcedureScheduler(727): Took xlock for pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_UPDATE_META; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=c09b5f7c44d4a83602090cbccc082b61, daughterA=25b84f3651d3c0adac7c0275b4cb1e0c, daughterB=205d5359b7ecb778c3d784ef1bb72d34 2023-04-12 02:55:41,746 INFO [Listener at localhost/46667] procedure.MasterProcedureScheduler(727): Took xlock for pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_UPDATE_META; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=c09b5f7c44d4a83602090cbccc082b61, daughterA=25b84f3651d3c0adac7c0275b4cb1e0c, daughterB=205d5359b7ecb778c3d784ef1bb72d34 2023-04-12 02:55:41,746 DEBUG [Listener at localhost/46667] procedure2.Procedure(946): pid=113, ppid=112, state=SUCCESS; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=c09b5f7c44d4a83602090cbccc082b61, UNASSIGN didn't hold the lock before restarting, skip acquiring lock. 2023-04-12 02:55:41,746 DEBUG [Listener at localhost/46667] procedure2.Procedure(946): pid=114, ppid=113, state=SUCCESS; CloseRegionProcedure c09b5f7c44d4a83602090cbccc082b61, server=jenkins-hbase4.apache.org,43091,1681268031804 didn't hold the lock before restarting, skip acquiring lock. 2023-04-12 02:55:41,746 INFO [Listener at localhost/46667] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 12 msec 2023-04-12 02:55:41,752 DEBUG [Listener at localhost/46667] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-04-12 02:55:41,753 DEBUG [Listener at localhost/46667] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase4.apache.org,44679,1681268031772, table=hbase:meta, region=1588230740 2023-04-12 02:55:41,754 INFO [Listener at localhost/46667] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1681268171754 2023-04-12 02:55:41,754 DEBUG [Listener at localhost/46667] assignment.AssignmentManager(1517): Joining cluster... 2023-04-12 02:55:41,756 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_UPDATE_META; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=c09b5f7c44d4a83602090cbccc082b61, daughterA=25b84f3651d3c0adac7c0275b4cb1e0c, daughterB=205d5359b7ecb778c3d784ef1bb72d34 2023-04-12 02:55:41,759 DEBUG [Listener at localhost/46667] assignment.RegionStateStore(147): Load hbase:meta entry region=1f255e980a31bb266f2206a30aa2ae3f, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,44679,1681268031772, regionLocation=jenkins-hbase4.apache.org,44679,1681268031772, openSeqNum=2 2023-04-12 02:55:41,759 DEBUG [Listener at localhost/46667] assignment.RegionStateStore(147): Load hbase:meta entry region=c09b5f7c44d4a83602090cbccc082b61, regionState=CLOSED, lastHost=jenkins-hbase4.apache.org,43091,1681268031804, regionLocation=jenkins-hbase4.apache.org,43091,1681268031804, openSeqNum=2 2023-04-12 02:55:41,759 INFO [Listener at localhost/46667] assignment.RegionStates(657): Added to offline, CURRENTLY NEVER CLEARED!!! state=CLOSED, location=jenkins-hbase4.apache.org,43091,1681268031804, table=testRecoveryAndDoubleExecution, region=c09b5f7c44d4a83602090cbccc082b61 2023-04-12 02:55:41,759 DEBUG [Listener at localhost/46667] assignment.RegionStateStore(147): Load hbase:meta entry region=25b84f3651d3c0adac7c0275b4cb1e0c, regionState=CLOSED, lastHost=null, regionLocation=null, openSeqNum=1 2023-04-12 02:55:41,759 DEBUG [Listener at localhost/46667] assignment.RegionStateStore(147): Load hbase:meta entry region=205d5359b7ecb778c3d784ef1bb72d34, regionState=CLOSED, lastHost=null, regionLocation=null, openSeqNum=1 2023-04-12 02:55:41,760 INFO [Listener at localhost/46667] assignment.AssignmentManager(1529): Number of RegionServers=3 2023-04-12 02:55:41,760 INFO [Listener at localhost/46667] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1681268201760 2023-04-12 02:55:41,760 INFO [Listener at localhost/46667] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1681268261760 2023-04-12 02:55:41,760 INFO [Listener at localhost/46667] assignment.AssignmentManager(1536): Joined the cluster in 5 msec 2023-04-12 02:55:41,760 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":3,"row":"testRecoveryAndDoubleExecution,,1681268136320.c09b5f7c44d4a83602090cbccc082b61.","families":{"info":[{"qualifier":"regioninfo","vlen":64,"tag":[],"timestamp":"1681268141760"},{"qualifier":"splitA","vlen":66,"tag":[],"timestamp":"1681268141760"},{"qualifier":"splitB","vlen":66,"tag":[],"timestamp":"1681268141760"}]},"ts":"1681268141760"} 2023-04-12 02:55:41,760 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":3,"row":"testRecoveryAndDoubleExecution,,1681268138347.25b84f3651d3c0adac7c0275b4cb1e0c.","families":{"info":[{"qualifier":"regioninfo","vlen":66,"tag":[],"timestamp":"1681268141760"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681268141760"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681268141760"}]},"ts":"1681268141760"} 2023-04-12 02:55:41,760 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":3,"row":"testRecoveryAndDoubleExecution,41,1681268138347.205d5359b7ecb778c3d784ef1bb72d34.","families":{"info":[{"qualifier":"regioninfo","vlen":66,"tag":[],"timestamp":"1681268141760"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681268141760"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681268141760"}]},"ts":"1681268141760"} 2023-04-12 02:55:41,762 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-04-12 02:55:41,765 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-04-12 02:55:41,765 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_OPEN_CHILD_REGIONS, locked=true; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=c09b5f7c44d4a83602090cbccc082b61, daughterA=25b84f3651d3c0adac7c0275b4cb1e0c, daughterB=205d5359b7ecb778c3d784ef1bb72d34 2023-04-12 02:55:41,765 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-04-12 02:55:41,765 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_OPEN_CHILD_REGIONS, locked=true; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=c09b5f7c44d4a83602090cbccc082b61, daughterA=25b84f3651d3c0adac7c0275b4cb1e0c, daughterB=205d5359b7ecb778c3d784ef1bb72d34 at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-04-12 02:55:42,010 INFO [Listener at localhost/46667] procedure.MasterProcedureTestingUtility(454): Restart 8 exec state=pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_OPEN_CHILD_REGIONS, locked=true; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=c09b5f7c44d4a83602090cbccc082b61, daughterA=25b84f3651d3c0adac7c0275b4cb1e0c, daughterB=205d5359b7ecb778c3d784ef1bb72d34 2023-04-12 02:55:42,010 INFO [Listener at localhost/46667] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-04-12 02:55:42,010 INFO [Listener at localhost/46667] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-04-12 02:55:42,010 INFO [Listener at localhost/46667] assignment.AssignmentManager(315): Stopping assignment manager 2023-04-12 02:55:42,011 ERROR [Listener at localhost/46667] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-04-12 02:55:42,011 INFO [Listener at localhost/46667] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-04-12 02:55:42,011 INFO [Listener at localhost/46667] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-04-12 02:55:42,011 INFO [Listener at localhost/46667] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-04-12 02:55:42,011 INFO [Listener at localhost/46667] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-04-12 02:55:42,012 INFO [Listener at localhost/46667] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-04-12 02:55:42,013 DEBUG [Listener at localhost/46667] store.ProcedureTree(114): Procedure Procedure(pid=112, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[0, 1, 2, 7, 8, 9, 10, 11] 2023-04-12 02:55:42,013 DEBUG [Listener at localhost/46667] store.ProcedureTree(114): Procedure Procedure(pid=113, ppid=112, class=org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure) stack ids=[3, 6] 2023-04-12 02:55:42,013 DEBUG [Listener at localhost/46667] store.ProcedureTree(114): Procedure Procedure(pid=114, ppid=113, class=org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure) stack ids=[4, 5] 2023-04-12 02:55:42,014 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-04-12 02:55:42,014 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-04-12 02:55:42,014 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-04-12 02:55:42,014 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-04-12 02:55:42,015 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-12 02:55:42,015 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=cc9012ec0b656baa24d04368fbe026c4, daughterA=e2a9b8fe15a781798c0b49e48ff5aad7, daughterB=6f55b5d6a460bd8c7569dee22f049104 2023-04-12 02:55:42,015 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-12 02:55:42,015 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-12 02:55:42,015 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-04-12 02:55:42,016 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=28, state=SUCCESS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=315e13e868a7f68a3ce37802b2f6312d, daughterA=4e1092f3c00484f8f09c730861225d43, daughterB=291734c00d212fd203b04cf8872d4649 2023-04-12 02:55:42,016 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=35, state=SUCCESS; DisableTableProcedure table=testSplitWithoutPONR 2023-04-12 02:55:42,016 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=40, state=SUCCESS; DeleteTableProcedure table=testSplitWithoutPONR 2023-04-12 02:55:42,016 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=41, state=SUCCESS; CreateTableProcedure table=testRollbackAndDoubleExecution 2023-04-12 02:55:42,017 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=44, state=ROLLEDBACK, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=96cdae2f7638212730b694522b9a464a, daughterA=fdd18e9cf539c13d863267ce16031583, daughterB=578095173fbc35e181cce14464037c2e 2023-04-12 02:55:42,017 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=47, state=SUCCESS; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=96cdae2f7638212730b694522b9a464a, ASSIGN 2023-04-12 02:55:42,017 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=49, state=SUCCESS; DisableTableProcedure table=testRollbackAndDoubleExecution 2023-04-12 02:55:42,017 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=52, state=SUCCESS; DeleteTableProcedure table=testRollbackAndDoubleExecution 2023-04-12 02:55:42,017 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=53, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionNoStoreFile 2023-04-12 02:55:42,018 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=56, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionNoStoreFile, parent=947433d660677a65ad9876ce2025e3e0, daughterA=b24cea6994eaa39f3128573197d02d9b, daughterB=b0e3d5fd7da33cb33084791956f0d7ca 2023-04-12 02:55:42,018 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=63, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionNoStoreFile 2023-04-12 02:55:42,018 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=68, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionNoStoreFile 2023-04-12 02:55:42,018 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=69, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:55:42,018 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=72, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionDeletedRowsDaughter, parent=1c6b8f18205322cbb248ea4fb10155d8, daughterA=81296e35c0d27ffa1ba7676ff53f87e1, daughterB=72b78dfd0678a920ddc3700612ff3932 2023-04-12 02:55:42,018 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=79, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:55:42,019 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=84, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:55:42,019 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=85, state=SUCCESS; CreateTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-04-12 02:55:42,019 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=88, state=SUCCESS; ModifyTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-04-12 02:55:42,019 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=95, state=ROLLEDBACK, exception=org.apache.hadoop.hbase.HBaseIOException via master-split-regions:org.apache.hadoop.hbase.HBaseIOException: The parent region state=OPENING, location=jenkins-hbase4.apache.org,34579,1681268031719, table=testRollbackForSplitTableRegionWithReplica, region=94855f8e18950d1c0c1e1d85df03f9ad is currently in transition, give up; SplitTableRegionProcedure table=testRollbackForSplitTableRegionWithReplica, parent=4381c9c8e57c5dbb2533908abaf07cf1, daughterA=2e35c2f6b8767f438f484c700a605299, daughterB=8407ddb6fc7704d121cd7f8a674aefb6 2023-04-12 02:55:42,019 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=96, state=SUCCESS; DisableTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-04-12 02:55:42,020 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=101, state=SUCCESS; DeleteTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-04-12 02:55:42,020 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=102, state=SUCCESS; CreateTableProcedure table=testInvalidSplitKey 2023-04-12 02:55:42,020 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=105, state=SUCCESS; DisableTableProcedure table=testInvalidSplitKey 2023-04-12 02:55:42,020 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=108, state=SUCCESS; DeleteTableProcedure table=testInvalidSplitKey 2023-04-12 02:55:42,020 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=109, state=SUCCESS; CreateTableProcedure table=testRecoveryAndDoubleExecution 2023-04-12 02:55:42,021 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(468): Loading pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_AFTER_META; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=c09b5f7c44d4a83602090cbccc082b61, daughterA=25b84f3651d3c0adac7c0275b4cb1e0c, daughterB=205d5359b7ecb778c3d784ef1bb72d34 2023-04-12 02:55:42,021 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(468): Loading pid=113, ppid=112, state=SUCCESS; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=c09b5f7c44d4a83602090cbccc082b61, UNASSIGN 2023-04-12 02:55:42,021 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(468): Loading pid=114, ppid=113, state=SUCCESS; CloseRegionProcedure c09b5f7c44d4a83602090cbccc082b61, server=jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:42,021 DEBUG [Listener at localhost/46667] procedure2.Procedure(966): pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_AFTER_META; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=c09b5f7c44d4a83602090cbccc082b61, daughterA=25b84f3651d3c0adac7c0275b4cb1e0c, daughterB=205d5359b7ecb778c3d784ef1bb72d34 held the lock before restarting, call acquireLock to restore it. 2023-04-12 02:55:42,021 INFO [Listener at localhost/46667] procedure.MasterProcedureScheduler(727): Took xlock for pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_AFTER_META; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=c09b5f7c44d4a83602090cbccc082b61, daughterA=25b84f3651d3c0adac7c0275b4cb1e0c, daughterB=205d5359b7ecb778c3d784ef1bb72d34 2023-04-12 02:55:42,021 INFO [Listener at localhost/46667] procedure.MasterProcedureScheduler(727): Took xlock for pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_AFTER_META; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=c09b5f7c44d4a83602090cbccc082b61, daughterA=25b84f3651d3c0adac7c0275b4cb1e0c, daughterB=205d5359b7ecb778c3d784ef1bb72d34 2023-04-12 02:55:42,021 INFO [Listener at localhost/46667] procedure.MasterProcedureScheduler(727): Took xlock for pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_AFTER_META; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=c09b5f7c44d4a83602090cbccc082b61, daughterA=25b84f3651d3c0adac7c0275b4cb1e0c, daughterB=205d5359b7ecb778c3d784ef1bb72d34 2023-04-12 02:55:42,021 DEBUG [Listener at localhost/46667] procedure2.Procedure(946): pid=113, ppid=112, state=SUCCESS; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=c09b5f7c44d4a83602090cbccc082b61, UNASSIGN didn't hold the lock before restarting, skip acquiring lock. 2023-04-12 02:55:42,022 DEBUG [Listener at localhost/46667] procedure2.Procedure(946): pid=114, ppid=113, state=SUCCESS; CloseRegionProcedure c09b5f7c44d4a83602090cbccc082b61, server=jenkins-hbase4.apache.org,43091,1681268031804 didn't hold the lock before restarting, skip acquiring lock. 2023-04-12 02:55:42,022 INFO [Listener at localhost/46667] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 10 msec 2023-04-12 02:55:42,022 DEBUG [Listener at localhost/46667] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-04-12 02:55:42,023 DEBUG [Listener at localhost/46667] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase4.apache.org,44679,1681268031772, table=hbase:meta, region=1588230740 2023-04-12 02:55:42,023 INFO [Listener at localhost/46667] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1681268172023 2023-04-12 02:55:42,023 DEBUG [Listener at localhost/46667] assignment.AssignmentManager(1517): Joining cluster... 2023-04-12 02:55:42,025 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_AFTER_META; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=c09b5f7c44d4a83602090cbccc082b61, daughterA=25b84f3651d3c0adac7c0275b4cb1e0c, daughterB=205d5359b7ecb778c3d784ef1bb72d34 2023-04-12 02:55:42,028 DEBUG [Listener at localhost/46667] assignment.RegionStateStore(147): Load hbase:meta entry region=1f255e980a31bb266f2206a30aa2ae3f, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,44679,1681268031772, regionLocation=jenkins-hbase4.apache.org,44679,1681268031772, openSeqNum=2 2023-04-12 02:55:42,028 DEBUG [Listener at localhost/46667] assignment.RegionStateStore(147): Load hbase:meta entry region=c09b5f7c44d4a83602090cbccc082b61, regionState=CLOSED, lastHost=jenkins-hbase4.apache.org,43091,1681268031804, regionLocation=jenkins-hbase4.apache.org,43091,1681268031804, openSeqNum=2 2023-04-12 02:55:42,028 INFO [Listener at localhost/46667] assignment.RegionStates(657): Added to offline, CURRENTLY NEVER CLEARED!!! state=CLOSED, location=jenkins-hbase4.apache.org,43091,1681268031804, table=testRecoveryAndDoubleExecution, region=c09b5f7c44d4a83602090cbccc082b61 2023-04-12 02:55:42,029 DEBUG [Listener at localhost/46667] assignment.RegionStateStore(147): Load hbase:meta entry region=25b84f3651d3c0adac7c0275b4cb1e0c, regionState=CLOSED, lastHost=null, regionLocation=null, openSeqNum=1 2023-04-12 02:55:42,029 DEBUG [Listener at localhost/46667] assignment.RegionStateStore(147): Load hbase:meta entry region=205d5359b7ecb778c3d784ef1bb72d34, regionState=CLOSED, lastHost=null, regionLocation=null, openSeqNum=1 2023-04-12 02:55:42,029 INFO [Listener at localhost/46667] assignment.AssignmentManager(1529): Number of RegionServers=3 2023-04-12 02:55:42,029 INFO [Listener at localhost/46667] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1681268202029 2023-04-12 02:55:42,029 INFO [Listener at localhost/46667] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1681268262029 2023-04-12 02:55:42,029 INFO [Listener at localhost/46667] assignment.AssignmentManager(1536): Joined the cluster in 5 msec 2023-04-12 02:55:42,029 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-04-12 02:55:42,032 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=115, ppid=112, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=25b84f3651d3c0adac7c0275b4cb1e0c, ASSIGN}, {pid=116, ppid=112, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=205d5359b7ecb778c3d784ef1bb72d34, ASSIGN}] 2023-04-12 02:55:42,032 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-04-12 02:55:42,032 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=112, state=WAITING:SPLIT_TABLE_REGION_POST_OPERATION, locked=true; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=c09b5f7c44d4a83602090cbccc082b61, daughterA=25b84f3651d3c0adac7c0275b4cb1e0c, daughterB=205d5359b7ecb778c3d784ef1bb72d34 2023-04-12 02:55:42,032 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-04-12 02:55:42,032 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=112, state=WAITING:SPLIT_TABLE_REGION_POST_OPERATION, locked=true; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=c09b5f7c44d4a83602090cbccc082b61, daughterA=25b84f3651d3c0adac7c0275b4cb1e0c, daughterB=205d5359b7ecb778c3d784ef1bb72d34 at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-04-12 02:55:42,251 ERROR [BP-1247239137-172.31.14.131-1681268026808 heartbeating to localhost/127.0.0.1:44691] datanode.BPServiceActor(848): Exception in BPOfferService for Block pool BP-1247239137-172.31.14.131-1681268026808 (Datanode Uuid 79922b64-644d-4757-a58c-9707ee400c82) service to localhost/127.0.0.1:44691 java.lang.IllegalThreadStateException at java.lang.ThreadGroup.addUnstarted(ThreadGroup.java:867) at java.lang.Thread.init(Thread.java:407) at java.lang.Thread.init(Thread.java:351) at java.lang.Thread.(Thread.java:501) at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService$1.newThread(FsDatasetAsyncDiskService.java:102) at java.util.concurrent.ThreadPoolExecutor$Worker.(ThreadPoolExecutor.java:619) at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:932) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1367) at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService.execute(FsDatasetAsyncDiskService.java:172) at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService.deleteAsync(FsDatasetAsyncDiskService.java:221) at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.invalidate(FsDatasetImpl.java:2216) at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.invalidate(FsDatasetImpl.java:2140) at org.apache.hadoop.hdfs.server.datanode.BPOfferService.processCommandFromActive(BPOfferService.java:732) at org.apache.hadoop.hdfs.server.datanode.BPOfferService.processCommandFromActor(BPOfferService.java:679) at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.processCommand(BPServiceActor.java:880) at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:676) at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:846) at java.lang.Thread.run(Thread.java:750) 2023-04-12 02:55:42,279 INFO [Listener at localhost/46667] procedure.MasterProcedureTestingUtility(454): Restart 9 exec state=pid=112, state=WAITING:SPLIT_TABLE_REGION_POST_OPERATION, locked=true; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=c09b5f7c44d4a83602090cbccc082b61, daughterA=25b84f3651d3c0adac7c0275b4cb1e0c, daughterB=205d5359b7ecb778c3d784ef1bb72d34 2023-04-12 02:55:42,280 INFO [Listener at localhost/46667] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-04-12 02:55:42,280 INFO [Listener at localhost/46667] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-04-12 02:55:42,280 INFO [Listener at localhost/46667] assignment.AssignmentManager(315): Stopping assignment manager 2023-04-12 02:55:42,280 ERROR [Listener at localhost/46667] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-04-12 02:55:42,280 INFO [Listener at localhost/46667] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-04-12 02:55:42,280 INFO [Listener at localhost/46667] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-04-12 02:55:42,280 INFO [Listener at localhost/46667] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-04-12 02:55:42,281 INFO [Listener at localhost/46667] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-04-12 02:55:42,281 INFO [Listener at localhost/46667] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-04-12 02:55:42,283 DEBUG [Listener at localhost/46667] store.ProcedureTree(114): Procedure Procedure(pid=112, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[0, 1, 2, 7, 8, 9, 10, 11, 12] 2023-04-12 02:55:42,283 DEBUG [Listener at localhost/46667] store.ProcedureTree(114): Procedure Procedure(pid=113, ppid=112, class=org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure) stack ids=[3, 6] 2023-04-12 02:55:42,283 DEBUG [Listener at localhost/46667] store.ProcedureTree(114): Procedure Procedure(pid=114, ppid=113, class=org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure) stack ids=[4, 5] 2023-04-12 02:55:42,283 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-04-12 02:55:42,284 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-04-12 02:55:42,284 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-04-12 02:55:42,284 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-04-12 02:55:42,284 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-12 02:55:42,284 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=cc9012ec0b656baa24d04368fbe026c4, daughterA=e2a9b8fe15a781798c0b49e48ff5aad7, daughterB=6f55b5d6a460bd8c7569dee22f049104 2023-04-12 02:55:42,285 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-12 02:55:42,285 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-12 02:55:42,285 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-04-12 02:55:42,285 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=28, state=SUCCESS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=315e13e868a7f68a3ce37802b2f6312d, daughterA=4e1092f3c00484f8f09c730861225d43, daughterB=291734c00d212fd203b04cf8872d4649 2023-04-12 02:55:42,285 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=35, state=SUCCESS; DisableTableProcedure table=testSplitWithoutPONR 2023-04-12 02:55:42,285 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=40, state=SUCCESS; DeleteTableProcedure table=testSplitWithoutPONR 2023-04-12 02:55:42,286 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=41, state=SUCCESS; CreateTableProcedure table=testRollbackAndDoubleExecution 2023-04-12 02:55:42,286 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=44, state=ROLLEDBACK, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=96cdae2f7638212730b694522b9a464a, daughterA=fdd18e9cf539c13d863267ce16031583, daughterB=578095173fbc35e181cce14464037c2e 2023-04-12 02:55:42,286 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=47, state=SUCCESS; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=96cdae2f7638212730b694522b9a464a, ASSIGN 2023-04-12 02:55:42,286 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=49, state=SUCCESS; DisableTableProcedure table=testRollbackAndDoubleExecution 2023-04-12 02:55:42,286 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=52, state=SUCCESS; DeleteTableProcedure table=testRollbackAndDoubleExecution 2023-04-12 02:55:42,287 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=53, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionNoStoreFile 2023-04-12 02:55:42,287 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=56, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionNoStoreFile, parent=947433d660677a65ad9876ce2025e3e0, daughterA=b24cea6994eaa39f3128573197d02d9b, daughterB=b0e3d5fd7da33cb33084791956f0d7ca 2023-04-12 02:55:42,287 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=63, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionNoStoreFile 2023-04-12 02:55:42,287 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=68, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionNoStoreFile 2023-04-12 02:55:42,287 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=69, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:55:42,288 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=72, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionDeletedRowsDaughter, parent=1c6b8f18205322cbb248ea4fb10155d8, daughterA=81296e35c0d27ffa1ba7676ff53f87e1, daughterB=72b78dfd0678a920ddc3700612ff3932 2023-04-12 02:55:42,288 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=79, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:55:42,288 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=84, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:55:42,288 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=85, state=SUCCESS; CreateTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-04-12 02:55:42,288 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=88, state=SUCCESS; ModifyTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-04-12 02:55:42,289 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=95, state=ROLLEDBACK, exception=org.apache.hadoop.hbase.HBaseIOException via master-split-regions:org.apache.hadoop.hbase.HBaseIOException: The parent region state=OPENING, location=jenkins-hbase4.apache.org,34579,1681268031719, table=testRollbackForSplitTableRegionWithReplica, region=94855f8e18950d1c0c1e1d85df03f9ad is currently in transition, give up; SplitTableRegionProcedure table=testRollbackForSplitTableRegionWithReplica, parent=4381c9c8e57c5dbb2533908abaf07cf1, daughterA=2e35c2f6b8767f438f484c700a605299, daughterB=8407ddb6fc7704d121cd7f8a674aefb6 2023-04-12 02:55:42,289 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=96, state=SUCCESS; DisableTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-04-12 02:55:42,289 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=101, state=SUCCESS; DeleteTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-04-12 02:55:42,289 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=102, state=SUCCESS; CreateTableProcedure table=testInvalidSplitKey 2023-04-12 02:55:42,289 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=105, state=SUCCESS; DisableTableProcedure table=testInvalidSplitKey 2023-04-12 02:55:42,289 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=108, state=SUCCESS; DeleteTableProcedure table=testInvalidSplitKey 2023-04-12 02:55:42,290 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=109, state=SUCCESS; CreateTableProcedure table=testRecoveryAndDoubleExecution 2023-04-12 02:55:42,290 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(468): Loading pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_OPEN_CHILD_REGIONS; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=c09b5f7c44d4a83602090cbccc082b61, daughterA=25b84f3651d3c0adac7c0275b4cb1e0c, daughterB=205d5359b7ecb778c3d784ef1bb72d34 2023-04-12 02:55:42,290 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(468): Loading pid=113, ppid=112, state=SUCCESS; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=c09b5f7c44d4a83602090cbccc082b61, UNASSIGN 2023-04-12 02:55:42,290 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(468): Loading pid=114, ppid=113, state=SUCCESS; CloseRegionProcedure c09b5f7c44d4a83602090cbccc082b61, server=jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:42,290 DEBUG [Listener at localhost/46667] procedure2.Procedure(966): pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_OPEN_CHILD_REGIONS; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=c09b5f7c44d4a83602090cbccc082b61, daughterA=25b84f3651d3c0adac7c0275b4cb1e0c, daughterB=205d5359b7ecb778c3d784ef1bb72d34 held the lock before restarting, call acquireLock to restore it. 2023-04-12 02:55:42,291 INFO [Listener at localhost/46667] procedure.MasterProcedureScheduler(727): Took xlock for pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_OPEN_CHILD_REGIONS; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=c09b5f7c44d4a83602090cbccc082b61, daughterA=25b84f3651d3c0adac7c0275b4cb1e0c, daughterB=205d5359b7ecb778c3d784ef1bb72d34 2023-04-12 02:55:42,291 INFO [Listener at localhost/46667] procedure.MasterProcedureScheduler(727): Took xlock for pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_OPEN_CHILD_REGIONS; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=c09b5f7c44d4a83602090cbccc082b61, daughterA=25b84f3651d3c0adac7c0275b4cb1e0c, daughterB=205d5359b7ecb778c3d784ef1bb72d34 2023-04-12 02:55:42,291 INFO [Listener at localhost/46667] procedure.MasterProcedureScheduler(727): Took xlock for pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_OPEN_CHILD_REGIONS; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=c09b5f7c44d4a83602090cbccc082b61, daughterA=25b84f3651d3c0adac7c0275b4cb1e0c, daughterB=205d5359b7ecb778c3d784ef1bb72d34 2023-04-12 02:55:42,291 DEBUG [Listener at localhost/46667] procedure2.Procedure(946): pid=113, ppid=112, state=SUCCESS; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=c09b5f7c44d4a83602090cbccc082b61, UNASSIGN didn't hold the lock before restarting, skip acquiring lock. 2023-04-12 02:55:42,291 DEBUG [Listener at localhost/46667] procedure2.Procedure(946): pid=114, ppid=113, state=SUCCESS; CloseRegionProcedure c09b5f7c44d4a83602090cbccc082b61, server=jenkins-hbase4.apache.org,43091,1681268031804 didn't hold the lock before restarting, skip acquiring lock. 2023-04-12 02:55:42,291 INFO [Listener at localhost/46667] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 9 msec 2023-04-12 02:55:42,292 DEBUG [Listener at localhost/46667] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-04-12 02:55:42,292 DEBUG [Listener at localhost/46667] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase4.apache.org,44679,1681268031772, table=hbase:meta, region=1588230740 2023-04-12 02:55:42,293 INFO [Listener at localhost/46667] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1681268172293 2023-04-12 02:55:42,294 DEBUG [Listener at localhost/46667] assignment.AssignmentManager(1517): Joining cluster... 2023-04-12 02:55:42,294 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_OPEN_CHILD_REGIONS; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=c09b5f7c44d4a83602090cbccc082b61, daughterA=25b84f3651d3c0adac7c0275b4cb1e0c, daughterB=205d5359b7ecb778c3d784ef1bb72d34 2023-04-12 02:55:42,299 DEBUG [Listener at localhost/46667] assignment.RegionStateStore(147): Load hbase:meta entry region=1f255e980a31bb266f2206a30aa2ae3f, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,44679,1681268031772, regionLocation=jenkins-hbase4.apache.org,44679,1681268031772, openSeqNum=2 2023-04-12 02:55:42,299 DEBUG [Listener at localhost/46667] assignment.RegionStateStore(147): Load hbase:meta entry region=c09b5f7c44d4a83602090cbccc082b61, regionState=CLOSED, lastHost=jenkins-hbase4.apache.org,43091,1681268031804, regionLocation=jenkins-hbase4.apache.org,43091,1681268031804, openSeqNum=2 2023-04-12 02:55:42,299 INFO [Listener at localhost/46667] assignment.RegionStates(657): Added to offline, CURRENTLY NEVER CLEARED!!! state=CLOSED, location=jenkins-hbase4.apache.org,43091,1681268031804, table=testRecoveryAndDoubleExecution, region=c09b5f7c44d4a83602090cbccc082b61 2023-04-12 02:55:42,299 DEBUG [Listener at localhost/46667] assignment.RegionStateStore(147): Load hbase:meta entry region=25b84f3651d3c0adac7c0275b4cb1e0c, regionState=CLOSED, lastHost=null, regionLocation=null, openSeqNum=1 2023-04-12 02:55:42,300 DEBUG [Listener at localhost/46667] assignment.RegionStateStore(147): Load hbase:meta entry region=205d5359b7ecb778c3d784ef1bb72d34, regionState=CLOSED, lastHost=null, regionLocation=null, openSeqNum=1 2023-04-12 02:55:42,300 INFO [Listener at localhost/46667] assignment.AssignmentManager(1529): Number of RegionServers=3 2023-04-12 02:55:42,300 INFO [Listener at localhost/46667] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1681268202300 2023-04-12 02:55:42,300 INFO [Listener at localhost/46667] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1681268262300 2023-04-12 02:55:42,300 INFO [Listener at localhost/46667] assignment.AssignmentManager(1536): Joined the cluster in 5 msec 2023-04-12 02:55:42,300 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=115, ppid=112, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=25b84f3651d3c0adac7c0275b4cb1e0c, ASSIGN}, {pid=116, ppid=112, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=205d5359b7ecb778c3d784ef1bb72d34, ASSIGN}] 2023-04-12 02:55:42,300 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-04-12 02:55:42,303 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=116, ppid=112, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=205d5359b7ecb778c3d784ef1bb72d34, ASSIGN 2023-04-12 02:55:42,303 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=116, ppid=112, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=205d5359b7ecb778c3d784ef1bb72d34, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,43091,1681268031804; forceNewPlan=false, retain=false 2023-04-12 02:55:42,304 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=115, ppid=112, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=25b84f3651d3c0adac7c0275b4cb1e0c, ASSIGN 2023-04-12 02:55:42,304 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=115, ppid=112, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=25b84f3651d3c0adac7c0275b4cb1e0c, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,43091,1681268031804; forceNewPlan=false, retain=false 2023-04-12 02:55:42,453 INFO [jenkins-hbase4:40281] balancer.BaseLoadBalancer(1545): Reassigned 2 regions. 2 retained the pre-restart assignment. 2023-04-12 02:55:42,455 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=116 updating hbase:meta row=205d5359b7ecb778c3d784ef1bb72d34, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:42,455 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRecoveryAndDoubleExecution,41,1681268138347.205d5359b7ecb778c3d784ef1bb72d34.","families":{"info":[{"qualifier":"regioninfo","vlen":66,"tag":[],"timestamp":"1681268142455"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681268142455"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681268142455"}]},"ts":"1681268142455"} 2023-04-12 02:55:42,457 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=117, ppid=116, state=RUNNABLE; OpenRegionProcedure 205d5359b7ecb778c3d784ef1bb72d34, server=jenkins-hbase4.apache.org,43091,1681268031804}] 2023-04-12 02:55:42,459 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=115 updating hbase:meta row=25b84f3651d3c0adac7c0275b4cb1e0c, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:42,459 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRecoveryAndDoubleExecution,,1681268138347.25b84f3651d3c0adac7c0275b4cb1e0c.","families":{"info":[{"qualifier":"regioninfo","vlen":66,"tag":[],"timestamp":"1681268142459"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681268142459"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681268142459"}]},"ts":"1681268142459"} 2023-04-12 02:55:42,460 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=118, ppid=115, state=RUNNABLE; OpenRegionProcedure 25b84f3651d3c0adac7c0275b4cb1e0c, server=jenkins-hbase4.apache.org,43091,1681268031804}] 2023-04-12 02:55:42,612 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRecoveryAndDoubleExecution,41,1681268138347.205d5359b7ecb778c3d784ef1bb72d34. 2023-04-12 02:55:42,612 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 205d5359b7ecb778c3d784ef1bb72d34, NAME => 'testRecoveryAndDoubleExecution,41,1681268138347.205d5359b7ecb778c3d784ef1bb72d34.', STARTKEY => '41', ENDKEY => ''} 2023-04-12 02:55:42,613 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(174): System coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro loaded, priority=536870911. 2023-04-12 02:55:42,613 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRecoveryAndDoubleExecution 205d5359b7ecb778c3d784ef1bb72d34 2023-04-12 02:55:42,613 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated testRecoveryAndDoubleExecution,41,1681268138347.205d5359b7ecb778c3d784ef1bb72d34.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-12 02:55:42,613 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 205d5359b7ecb778c3d784ef1bb72d34 2023-04-12 02:55:42,613 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 205d5359b7ecb778c3d784ef1bb72d34 2023-04-12 02:55:42,614 INFO [StoreOpener-205d5359b7ecb778c3d784ef1bb72d34-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf1 of region 205d5359b7ecb778c3d784ef1bb72d34 2023-04-12 02:55:42,615 DEBUG [StoreOpener-205d5359b7ecb778c3d784ef1bb72d34-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/205d5359b7ecb778c3d784ef1bb72d34/cf1 2023-04-12 02:55:42,615 DEBUG [StoreOpener-205d5359b7ecb778c3d784ef1bb72d34-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/205d5359b7ecb778c3d784ef1bb72d34/cf1 2023-04-12 02:55:42,616 INFO [StoreOpener-205d5359b7ecb778c3d784ef1bb72d34-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 205d5359b7ecb778c3d784ef1bb72d34 columnFamilyName cf1 2023-04-12 02:55:42,639 DEBUG [StoreOpener-205d5359b7ecb778c3d784ef1bb72d34-1] regionserver.HStore(539): loaded hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/205d5359b7ecb778c3d784ef1bb72d34/cf1/001e134c60444ac2b388f9df6bf5670b.c09b5f7c44d4a83602090cbccc082b61->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf1/001e134c60444ac2b388f9df6bf5670b-top 2023-04-12 02:55:42,643 DEBUG [StoreOpener-205d5359b7ecb778c3d784ef1bb72d34-1] regionserver.HStore(539): loaded hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/205d5359b7ecb778c3d784ef1bb72d34/cf1/1373dcc8adeb4ae4b9929ef46fbb4980.c09b5f7c44d4a83602090cbccc082b61->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf1/1373dcc8adeb4ae4b9929ef46fbb4980-top 2023-04-12 02:55:42,655 DEBUG [StoreOpener-205d5359b7ecb778c3d784ef1bb72d34-1] regionserver.HStore(539): loaded hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/205d5359b7ecb778c3d784ef1bb72d34/cf1/63025bd542924f2caced3f22c416a5a1.c09b5f7c44d4a83602090cbccc082b61->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf1/63025bd542924f2caced3f22c416a5a1-top 2023-04-12 02:55:42,659 DEBUG [StoreOpener-205d5359b7ecb778c3d784ef1bb72d34-1] regionserver.HStore(539): loaded hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/205d5359b7ecb778c3d784ef1bb72d34/cf1/9d5ec6ededfd4de8b6d44e635d18346c.c09b5f7c44d4a83602090cbccc082b61->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf1/9d5ec6ededfd4de8b6d44e635d18346c-top 2023-04-12 02:55:42,667 DEBUG [StoreOpener-205d5359b7ecb778c3d784ef1bb72d34-1] regionserver.HStore(539): loaded hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/205d5359b7ecb778c3d784ef1bb72d34/cf1/a79c7cc5c59c4f6bbe6a6a584d824f36.c09b5f7c44d4a83602090cbccc082b61->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf1/a79c7cc5c59c4f6bbe6a6a584d824f36-top 2023-04-12 02:55:42,672 DEBUG [StoreOpener-205d5359b7ecb778c3d784ef1bb72d34-1] regionserver.HStore(539): loaded hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/205d5359b7ecb778c3d784ef1bb72d34/cf1/c14bd2fe66004d6d842c72f4b2a4ee90.c09b5f7c44d4a83602090cbccc082b61->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf1/c14bd2fe66004d6d842c72f4b2a4ee90-top 2023-04-12 02:55:42,676 DEBUG [StoreOpener-205d5359b7ecb778c3d784ef1bb72d34-1] regionserver.HStore(539): loaded hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/205d5359b7ecb778c3d784ef1bb72d34/cf1/e62dda4fe11148e88206bdd397ca64c0.c09b5f7c44d4a83602090cbccc082b61->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf1/e62dda4fe11148e88206bdd397ca64c0-top 2023-04-12 02:55:42,676 INFO [StoreOpener-205d5359b7ecb778c3d784ef1bb72d34-1] regionserver.HStore(310): Store=205d5359b7ecb778c3d784ef1bb72d34/cf1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-12 02:55:42,677 INFO [StoreOpener-205d5359b7ecb778c3d784ef1bb72d34-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf2 of region 205d5359b7ecb778c3d784ef1bb72d34 2023-04-12 02:55:42,677 DEBUG [StoreOpener-205d5359b7ecb778c3d784ef1bb72d34-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/205d5359b7ecb778c3d784ef1bb72d34/cf2 2023-04-12 02:55:42,678 DEBUG [StoreOpener-205d5359b7ecb778c3d784ef1bb72d34-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/205d5359b7ecb778c3d784ef1bb72d34/cf2 2023-04-12 02:55:42,678 INFO [StoreOpener-205d5359b7ecb778c3d784ef1bb72d34-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 205d5359b7ecb778c3d784ef1bb72d34 columnFamilyName cf2 2023-04-12 02:55:42,697 DEBUG [StoreOpener-205d5359b7ecb778c3d784ef1bb72d34-1] regionserver.HStore(539): loaded hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/205d5359b7ecb778c3d784ef1bb72d34/cf2/1b3fc837d85449cba8b914aa7f18f0fe.c09b5f7c44d4a83602090cbccc082b61->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf2/1b3fc837d85449cba8b914aa7f18f0fe-top 2023-04-12 02:55:42,703 DEBUG [StoreOpener-205d5359b7ecb778c3d784ef1bb72d34-1] regionserver.HStore(539): loaded hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/205d5359b7ecb778c3d784ef1bb72d34/cf2/5b22fe6156a04c8c9486fd6f189bf4d7.c09b5f7c44d4a83602090cbccc082b61->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf2/5b22fe6156a04c8c9486fd6f189bf4d7-top 2023-04-12 02:55:42,709 DEBUG [StoreOpener-205d5359b7ecb778c3d784ef1bb72d34-1] regionserver.HStore(539): loaded hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/205d5359b7ecb778c3d784ef1bb72d34/cf2/a4e6a82328b24916838ccb970ddd7704.c09b5f7c44d4a83602090cbccc082b61->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf2/a4e6a82328b24916838ccb970ddd7704-top 2023-04-12 02:55:42,716 DEBUG [StoreOpener-205d5359b7ecb778c3d784ef1bb72d34-1] regionserver.HStore(539): loaded hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/205d5359b7ecb778c3d784ef1bb72d34/cf2/a7d4784e93134ef2943643862fd08463.c09b5f7c44d4a83602090cbccc082b61->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf2/a7d4784e93134ef2943643862fd08463-top 2023-04-12 02:55:42,721 DEBUG [StoreOpener-205d5359b7ecb778c3d784ef1bb72d34-1] regionserver.HStore(539): loaded hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/205d5359b7ecb778c3d784ef1bb72d34/cf2/f70a7ebc6c464a08ab00a25e137bbc72.c09b5f7c44d4a83602090cbccc082b61->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf2/f70a7ebc6c464a08ab00a25e137bbc72-top 2023-04-12 02:55:42,726 DEBUG [StoreOpener-205d5359b7ecb778c3d784ef1bb72d34-1] regionserver.HStore(539): loaded hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/205d5359b7ecb778c3d784ef1bb72d34/cf2/fb8ce8dc74bd47fabd120f6f688962d5.c09b5f7c44d4a83602090cbccc082b61->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf2/fb8ce8dc74bd47fabd120f6f688962d5-top 2023-04-12 02:55:42,730 DEBUG [StoreOpener-205d5359b7ecb778c3d784ef1bb72d34-1] regionserver.HStore(539): loaded hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/205d5359b7ecb778c3d784ef1bb72d34/cf2/fe343f29642942069fb8e38396f28358.c09b5f7c44d4a83602090cbccc082b61->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf2/fe343f29642942069fb8e38396f28358-top 2023-04-12 02:55:42,730 INFO [StoreOpener-205d5359b7ecb778c3d784ef1bb72d34-1] regionserver.HStore(310): Store=205d5359b7ecb778c3d784ef1bb72d34/cf2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-12 02:55:42,731 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/205d5359b7ecb778c3d784ef1bb72d34 2023-04-12 02:55:42,733 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/205d5359b7ecb778c3d784ef1bb72d34 2023-04-12 02:55:42,734 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testRecoveryAndDoubleExecution descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-04-12 02:55:42,736 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 205d5359b7ecb778c3d784ef1bb72d34 2023-04-12 02:55:42,737 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 205d5359b7ecb778c3d784ef1bb72d34; next sequenceid=86; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=69514451, jitterRate=0.0358460396528244}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-04-12 02:55:42,737 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 205d5359b7ecb778c3d784ef1bb72d34: 2023-04-12 02:55:42,738 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRecoveryAndDoubleExecution,41,1681268138347.205d5359b7ecb778c3d784ef1bb72d34., pid=117, masterSystemTime=1681268142608 2023-04-12 02:55:42,738 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: Opening Region; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 02:55:42,738 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: Opening Region; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 02:55:42,739 DEBUG [RS:2;jenkins-hbase4:43091-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-12 02:55:42,739 DEBUG [RS:2;jenkins-hbase4:43091-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-12 02:55:42,739 INFO [RS:2;jenkins-hbase4:43091-longCompactions-0] regionserver.HStore(1898): Keeping/Overriding Compaction request priority to -2147482648 for CF cf2 since it belongs to recently split daughter region testRecoveryAndDoubleExecution,41,1681268138347.205d5359b7ecb778c3d784ef1bb72d34. 2023-04-12 02:55:42,739 DEBUG [RS:2;jenkins-hbase4:43091-longCompactions-0] regionserver.HStore(1912): 205d5359b7ecb778c3d784ef1bb72d34/cf2 is initiating minor compaction (all files) 2023-04-12 02:55:42,740 INFO [RS:2;jenkins-hbase4:43091-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 205d5359b7ecb778c3d784ef1bb72d34/cf2 in testRecoveryAndDoubleExecution,41,1681268138347.205d5359b7ecb778c3d784ef1bb72d34. 2023-04-12 02:55:42,740 INFO [RS:2;jenkins-hbase4:43091-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/205d5359b7ecb778c3d784ef1bb72d34/cf2/5b22fe6156a04c8c9486fd6f189bf4d7.c09b5f7c44d4a83602090cbccc082b61->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf2/5b22fe6156a04c8c9486fd6f189bf4d7-top, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/205d5359b7ecb778c3d784ef1bb72d34/cf2/a4e6a82328b24916838ccb970ddd7704.c09b5f7c44d4a83602090cbccc082b61->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf2/a4e6a82328b24916838ccb970ddd7704-top, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/205d5359b7ecb778c3d784ef1bb72d34/cf2/fb8ce8dc74bd47fabd120f6f688962d5.c09b5f7c44d4a83602090cbccc082b61->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf2/fb8ce8dc74bd47fabd120f6f688962d5-top, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/205d5359b7ecb778c3d784ef1bb72d34/cf2/fe343f29642942069fb8e38396f28358.c09b5f7c44d4a83602090cbccc082b61->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf2/fe343f29642942069fb8e38396f28358-top, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/205d5359b7ecb778c3d784ef1bb72d34/cf2/1b3fc837d85449cba8b914aa7f18f0fe.c09b5f7c44d4a83602090cbccc082b61->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf2/1b3fc837d85449cba8b914aa7f18f0fe-top, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/205d5359b7ecb778c3d784ef1bb72d34/cf2/a7d4784e93134ef2943643862fd08463.c09b5f7c44d4a83602090cbccc082b61->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf2/a7d4784e93134ef2943643862fd08463-top, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/205d5359b7ecb778c3d784ef1bb72d34/cf2/f70a7ebc6c464a08ab00a25e137bbc72.c09b5f7c44d4a83602090cbccc082b61->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf2/f70a7ebc6c464a08ab00a25e137bbc72-top] into tmpdir=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/205d5359b7ecb778c3d784ef1bb72d34/.tmp, totalSize=34.9 K 2023-04-12 02:55:42,741 INFO [RS:2;jenkins-hbase4:43091-shortCompactions-0] regionserver.HStore(1898): Keeping/Overriding Compaction request priority to -2147482648 for CF cf1 since it belongs to recently split daughter region testRecoveryAndDoubleExecution,41,1681268138347.205d5359b7ecb778c3d784ef1bb72d34. 2023-04-12 02:55:42,741 DEBUG [RS:2;jenkins-hbase4:43091-longCompactions-0] compactions.Compactor(207): Compacting 5b22fe6156a04c8c9486fd6f189bf4d7.c09b5f7c44d4a83602090cbccc082b61, keycount=1, bloomtype=ROW, size=4.8 K, encoding=NONE, compression=NONE, seqNum=7, earliestPutTs=1681268136828 2023-04-12 02:55:42,741 DEBUG [RS:2;jenkins-hbase4:43091-shortCompactions-0] regionserver.HStore(1912): 205d5359b7ecb778c3d784ef1bb72d34/cf1 is initiating minor compaction (all files) 2023-04-12 02:55:42,741 INFO [RS:2;jenkins-hbase4:43091-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 205d5359b7ecb778c3d784ef1bb72d34/cf1 in testRecoveryAndDoubleExecution,41,1681268138347.205d5359b7ecb778c3d784ef1bb72d34. 2023-04-12 02:55:42,741 DEBUG [RS:2;jenkins-hbase4:43091-longCompactions-0] compactions.Compactor(207): Compacting a4e6a82328b24916838ccb970ddd7704.c09b5f7c44d4a83602090cbccc082b61, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=20, earliestPutTs=1681268137049 2023-04-12 02:55:42,741 INFO [RS:2;jenkins-hbase4:43091-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/205d5359b7ecb778c3d784ef1bb72d34/cf1/a79c7cc5c59c4f6bbe6a6a584d824f36.c09b5f7c44d4a83602090cbccc082b61->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf1/a79c7cc5c59c4f6bbe6a6a584d824f36-top, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/205d5359b7ecb778c3d784ef1bb72d34/cf1/63025bd542924f2caced3f22c416a5a1.c09b5f7c44d4a83602090cbccc082b61->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf1/63025bd542924f2caced3f22c416a5a1-top, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/205d5359b7ecb778c3d784ef1bb72d34/cf1/9d5ec6ededfd4de8b6d44e635d18346c.c09b5f7c44d4a83602090cbccc082b61->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf1/9d5ec6ededfd4de8b6d44e635d18346c-top, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/205d5359b7ecb778c3d784ef1bb72d34/cf1/001e134c60444ac2b388f9df6bf5670b.c09b5f7c44d4a83602090cbccc082b61->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf1/001e134c60444ac2b388f9df6bf5670b-top, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/205d5359b7ecb778c3d784ef1bb72d34/cf1/c14bd2fe66004d6d842c72f4b2a4ee90.c09b5f7c44d4a83602090cbccc082b61->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf1/c14bd2fe66004d6d842c72f4b2a4ee90-top, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/205d5359b7ecb778c3d784ef1bb72d34/cf1/1373dcc8adeb4ae4b9929ef46fbb4980.c09b5f7c44d4a83602090cbccc082b61->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf1/1373dcc8adeb4ae4b9929ef46fbb4980-top, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/205d5359b7ecb778c3d784ef1bb72d34/cf1/e62dda4fe11148e88206bdd397ca64c0.c09b5f7c44d4a83602090cbccc082b61->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf1/e62dda4fe11148e88206bdd397ca64c0-top] into tmpdir=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/205d5359b7ecb778c3d784ef1bb72d34/.tmp, totalSize=34.9 K 2023-04-12 02:55:42,742 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRecoveryAndDoubleExecution,41,1681268138347.205d5359b7ecb778c3d784ef1bb72d34. 2023-04-12 02:55:42,742 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRecoveryAndDoubleExecution,41,1681268138347.205d5359b7ecb778c3d784ef1bb72d34. 2023-04-12 02:55:42,742 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRecoveryAndDoubleExecution,,1681268138347.25b84f3651d3c0adac7c0275b4cb1e0c. 2023-04-12 02:55:42,742 DEBUG [RS:2;jenkins-hbase4:43091-longCompactions-0] compactions.Compactor(207): Compacting fb8ce8dc74bd47fabd120f6f688962d5.c09b5f7c44d4a83602090cbccc082b61, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=33, earliestPutTs=1681268137249 2023-04-12 02:55:42,742 DEBUG [RS:2;jenkins-hbase4:43091-shortCompactions-0] compactions.Compactor(207): Compacting a79c7cc5c59c4f6bbe6a6a584d824f36.c09b5f7c44d4a83602090cbccc082b61, keycount=1, bloomtype=ROW, size=4.8 K, encoding=NONE, compression=NONE, seqNum=7, earliestPutTs=1681268136828 2023-04-12 02:55:42,742 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 25b84f3651d3c0adac7c0275b4cb1e0c, NAME => 'testRecoveryAndDoubleExecution,,1681268138347.25b84f3651d3c0adac7c0275b4cb1e0c.', STARTKEY => '', ENDKEY => '41'} 2023-04-12 02:55:42,742 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(174): System coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro loaded, priority=536870911. 2023-04-12 02:55:42,743 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRecoveryAndDoubleExecution 25b84f3651d3c0adac7c0275b4cb1e0c 2023-04-12 02:55:42,743 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated testRecoveryAndDoubleExecution,,1681268138347.25b84f3651d3c0adac7c0275b4cb1e0c.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-12 02:55:42,743 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 25b84f3651d3c0adac7c0275b4cb1e0c 2023-04-12 02:55:42,743 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 25b84f3651d3c0adac7c0275b4cb1e0c 2023-04-12 02:55:42,743 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=116 updating hbase:meta row=205d5359b7ecb778c3d784ef1bb72d34, regionState=OPEN, openSeqNum=86, regionLocation=jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:42,744 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRecoveryAndDoubleExecution,41,1681268138347.205d5359b7ecb778c3d784ef1bb72d34.","families":{"info":[{"qualifier":"regioninfo","vlen":66,"tag":[],"timestamp":"1681268142743"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1681268142743"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681268142743"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681268142743"}]},"ts":"1681268142743"} 2023-04-12 02:55:42,744 DEBUG [RS:2;jenkins-hbase4:43091-longCompactions-0] compactions.Compactor(207): Compacting fe343f29642942069fb8e38396f28358.c09b5f7c44d4a83602090cbccc082b61, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=46, earliestPutTs=1681268137466 2023-04-12 02:55:42,746 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=117, resume processing ppid=116 2023-04-12 02:55:42,746 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=117, ppid=116, state=SUCCESS; OpenRegionProcedure 205d5359b7ecb778c3d784ef1bb72d34, server=jenkins-hbase4.apache.org,43091,1681268031804 in 288 msec 2023-04-12 02:55:42,748 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=116, ppid=112, state=SUCCESS; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=205d5359b7ecb778c3d784ef1bb72d34, ASSIGN in 447 msec 2023-04-12 02:55:42,748 DEBUG [RS:2;jenkins-hbase4:43091-shortCompactions-0] compactions.Compactor(207): Compacting 63025bd542924f2caced3f22c416a5a1.c09b5f7c44d4a83602090cbccc082b61, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=20, earliestPutTs=1681268137049 2023-04-12 02:55:42,748 DEBUG [RS:2;jenkins-hbase4:43091-longCompactions-0] compactions.Compactor(207): Compacting 1b3fc837d85449cba8b914aa7f18f0fe.c09b5f7c44d4a83602090cbccc082b61, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=59, earliestPutTs=1681268137658 2023-04-12 02:55:42,749 INFO [StoreOpener-25b84f3651d3c0adac7c0275b4cb1e0c-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf1 of region 25b84f3651d3c0adac7c0275b4cb1e0c 2023-04-12 02:55:42,749 DEBUG [RS:2;jenkins-hbase4:43091-longCompactions-0] compactions.Compactor(207): Compacting a7d4784e93134ef2943643862fd08463.c09b5f7c44d4a83602090cbccc082b61, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=72, earliestPutTs=1681268137880 2023-04-12 02:55:42,749 DEBUG [RS:2;jenkins-hbase4:43091-shortCompactions-0] compactions.Compactor(207): Compacting 9d5ec6ededfd4de8b6d44e635d18346c.c09b5f7c44d4a83602090cbccc082b61, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=33, earliestPutTs=1681268137249 2023-04-12 02:55:42,750 DEBUG [RS:2;jenkins-hbase4:43091-longCompactions-0] compactions.Compactor(207): Compacting f70a7ebc6c464a08ab00a25e137bbc72.c09b5f7c44d4a83602090cbccc082b61, keycount=4, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=83, earliestPutTs=1681268138088 2023-04-12 02:55:42,750 DEBUG [RS:2;jenkins-hbase4:43091-shortCompactions-0] compactions.Compactor(207): Compacting 001e134c60444ac2b388f9df6bf5670b.c09b5f7c44d4a83602090cbccc082b61, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=46, earliestPutTs=1681268137466 2023-04-12 02:55:42,751 DEBUG [RS:2;jenkins-hbase4:43091-shortCompactions-0] compactions.Compactor(207): Compacting c14bd2fe66004d6d842c72f4b2a4ee90.c09b5f7c44d4a83602090cbccc082b61, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=59, earliestPutTs=1681268137658 2023-04-12 02:55:42,751 DEBUG [RS:2;jenkins-hbase4:43091-shortCompactions-0] compactions.Compactor(207): Compacting 1373dcc8adeb4ae4b9929ef46fbb4980.c09b5f7c44d4a83602090cbccc082b61, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=72, earliestPutTs=1681268137880 2023-04-12 02:55:42,752 DEBUG [RS:2;jenkins-hbase4:43091-shortCompactions-0] compactions.Compactor(207): Compacting e62dda4fe11148e88206bdd397ca64c0.c09b5f7c44d4a83602090cbccc082b61, keycount=4, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=83, earliestPutTs=1681268138088 2023-04-12 02:55:42,760 DEBUG [StoreOpener-25b84f3651d3c0adac7c0275b4cb1e0c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/25b84f3651d3c0adac7c0275b4cb1e0c/cf1 2023-04-12 02:55:42,760 DEBUG [StoreOpener-25b84f3651d3c0adac7c0275b4cb1e0c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/25b84f3651d3c0adac7c0275b4cb1e0c/cf1 2023-04-12 02:55:42,761 INFO [StoreOpener-25b84f3651d3c0adac7c0275b4cb1e0c-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 25b84f3651d3c0adac7c0275b4cb1e0c columnFamilyName cf1 2023-04-12 02:55:42,764 INFO [RS:2;jenkins-hbase4:43091-longCompactions-0] throttle.PressureAwareThroughputController(145): 205d5359b7ecb778c3d784ef1bb72d34#cf2#compaction#108 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 02:55:42,788 DEBUG [StoreOpener-25b84f3651d3c0adac7c0275b4cb1e0c-1] regionserver.HStore(539): loaded hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/25b84f3651d3c0adac7c0275b4cb1e0c/cf1/001e134c60444ac2b388f9df6bf5670b.c09b5f7c44d4a83602090cbccc082b61->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf1/001e134c60444ac2b388f9df6bf5670b-bottom 2023-04-12 02:55:42,791 DEBUG [StoreOpener-25b84f3651d3c0adac7c0275b4cb1e0c-1] regionserver.HStore(539): loaded hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/25b84f3651d3c0adac7c0275b4cb1e0c/cf1/1373dcc8adeb4ae4b9929ef46fbb4980.c09b5f7c44d4a83602090cbccc082b61->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf1/1373dcc8adeb4ae4b9929ef46fbb4980-bottom 2023-04-12 02:55:42,796 DEBUG [StoreOpener-25b84f3651d3c0adac7c0275b4cb1e0c-1] regionserver.HStore(539): loaded hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/25b84f3651d3c0adac7c0275b4cb1e0c/cf1/63025bd542924f2caced3f22c416a5a1.c09b5f7c44d4a83602090cbccc082b61->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf1/63025bd542924f2caced3f22c416a5a1-bottom 2023-04-12 02:55:42,796 INFO [RS:2;jenkins-hbase4:43091-shortCompactions-0] throttle.PressureAwareThroughputController(145): 205d5359b7ecb778c3d784ef1bb72d34#cf1#compaction#109 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 02:55:42,801 DEBUG [StoreOpener-25b84f3651d3c0adac7c0275b4cb1e0c-1] regionserver.HStore(539): loaded hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/25b84f3651d3c0adac7c0275b4cb1e0c/cf1/9d5ec6ededfd4de8b6d44e635d18346c.c09b5f7c44d4a83602090cbccc082b61->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf1/9d5ec6ededfd4de8b6d44e635d18346c-bottom 2023-04-12 02:55:42,806 DEBUG [RS:2;jenkins-hbase4:43091-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/205d5359b7ecb778c3d784ef1bb72d34/.tmp/cf2/df99aec444e54dbab97c30b0d5f46460 as hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/205d5359b7ecb778c3d784ef1bb72d34/cf2/df99aec444e54dbab97c30b0d5f46460 2023-04-12 02:55:42,810 DEBUG [StoreOpener-25b84f3651d3c0adac7c0275b4cb1e0c-1] regionserver.HStore(539): loaded hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/25b84f3651d3c0adac7c0275b4cb1e0c/cf1/a79c7cc5c59c4f6bbe6a6a584d824f36.c09b5f7c44d4a83602090cbccc082b61->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf1/a79c7cc5c59c4f6bbe6a6a584d824f36-bottom 2023-04-12 02:55:42,817 INFO [RS:2;jenkins-hbase4:43091-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 205d5359b7ecb778c3d784ef1bb72d34/cf2 of 205d5359b7ecb778c3d784ef1bb72d34 into df99aec444e54dbab97c30b0d5f46460(size=6.1 K), total size for store is 6.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 02:55:42,818 DEBUG [RS:2;jenkins-hbase4:43091-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 205d5359b7ecb778c3d784ef1bb72d34: 2023-04-12 02:55:42,818 INFO [RS:2;jenkins-hbase4:43091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=testRecoveryAndDoubleExecution,41,1681268138347.205d5359b7ecb778c3d784ef1bb72d34., storeName=205d5359b7ecb778c3d784ef1bb72d34/cf2, priority=9, startTime=1681268142738; duration=0sec 2023-04-12 02:55:42,818 DEBUG [RS:2;jenkins-hbase4:43091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 02:55:42,824 DEBUG [StoreOpener-25b84f3651d3c0adac7c0275b4cb1e0c-1] regionserver.HStore(539): loaded hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/25b84f3651d3c0adac7c0275b4cb1e0c/cf1/c14bd2fe66004d6d842c72f4b2a4ee90.c09b5f7c44d4a83602090cbccc082b61->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf1/c14bd2fe66004d6d842c72f4b2a4ee90-bottom 2023-04-12 02:55:42,827 DEBUG [StoreOpener-25b84f3651d3c0adac7c0275b4cb1e0c-1] regionserver.HStore(539): loaded hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/25b84f3651d3c0adac7c0275b4cb1e0c/cf1/e62dda4fe11148e88206bdd397ca64c0.c09b5f7c44d4a83602090cbccc082b61->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf1/e62dda4fe11148e88206bdd397ca64c0-bottom 2023-04-12 02:55:42,827 INFO [StoreOpener-25b84f3651d3c0adac7c0275b4cb1e0c-1] regionserver.HStore(310): Store=25b84f3651d3c0adac7c0275b4cb1e0c/cf1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-12 02:55:42,827 INFO [StoreOpener-25b84f3651d3c0adac7c0275b4cb1e0c-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf2 of region 25b84f3651d3c0adac7c0275b4cb1e0c 2023-04-12 02:55:42,829 DEBUG [StoreOpener-25b84f3651d3c0adac7c0275b4cb1e0c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/25b84f3651d3c0adac7c0275b4cb1e0c/cf2 2023-04-12 02:55:42,829 DEBUG [StoreOpener-25b84f3651d3c0adac7c0275b4cb1e0c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/25b84f3651d3c0adac7c0275b4cb1e0c/cf2 2023-04-12 02:55:42,829 INFO [StoreOpener-25b84f3651d3c0adac7c0275b4cb1e0c-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 25b84f3651d3c0adac7c0275b4cb1e0c columnFamilyName cf2 2023-04-12 02:55:42,843 DEBUG [RS:2;jenkins-hbase4:43091-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/205d5359b7ecb778c3d784ef1bb72d34/.tmp/cf1/2f03e2dbb8c9448b8297606a5f3ae63b as hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/205d5359b7ecb778c3d784ef1bb72d34/cf1/2f03e2dbb8c9448b8297606a5f3ae63b 2023-04-12 02:55:42,847 INFO [RS:2;jenkins-hbase4:43091-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 205d5359b7ecb778c3d784ef1bb72d34/cf1 of 205d5359b7ecb778c3d784ef1bb72d34 into 2f03e2dbb8c9448b8297606a5f3ae63b(size=6.1 K), total size for store is 6.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 02:55:42,847 DEBUG [RS:2;jenkins-hbase4:43091-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 205d5359b7ecb778c3d784ef1bb72d34: 2023-04-12 02:55:42,847 INFO [RS:2;jenkins-hbase4:43091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=testRecoveryAndDoubleExecution,41,1681268138347.205d5359b7ecb778c3d784ef1bb72d34., storeName=205d5359b7ecb778c3d784ef1bb72d34/cf1, priority=9, startTime=1681268142738; duration=0sec 2023-04-12 02:55:42,847 DEBUG [RS:2;jenkins-hbase4:43091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 02:55:42,851 DEBUG [StoreOpener-25b84f3651d3c0adac7c0275b4cb1e0c-1] regionserver.HStore(539): loaded hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/25b84f3651d3c0adac7c0275b4cb1e0c/cf2/1b3fc837d85449cba8b914aa7f18f0fe.c09b5f7c44d4a83602090cbccc082b61->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf2/1b3fc837d85449cba8b914aa7f18f0fe-bottom 2023-04-12 02:55:42,853 DEBUG [StoreOpener-25b84f3651d3c0adac7c0275b4cb1e0c-1] regionserver.HStore(539): loaded hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/25b84f3651d3c0adac7c0275b4cb1e0c/cf2/5b22fe6156a04c8c9486fd6f189bf4d7.c09b5f7c44d4a83602090cbccc082b61->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf2/5b22fe6156a04c8c9486fd6f189bf4d7-bottom 2023-04-12 02:55:42,856 DEBUG [StoreOpener-25b84f3651d3c0adac7c0275b4cb1e0c-1] regionserver.HStore(539): loaded hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/25b84f3651d3c0adac7c0275b4cb1e0c/cf2/a4e6a82328b24916838ccb970ddd7704.c09b5f7c44d4a83602090cbccc082b61->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf2/a4e6a82328b24916838ccb970ddd7704-bottom 2023-04-12 02:55:42,864 DEBUG [StoreOpener-25b84f3651d3c0adac7c0275b4cb1e0c-1] regionserver.HStore(539): loaded hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/25b84f3651d3c0adac7c0275b4cb1e0c/cf2/a7d4784e93134ef2943643862fd08463.c09b5f7c44d4a83602090cbccc082b61->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf2/a7d4784e93134ef2943643862fd08463-bottom 2023-04-12 02:55:42,867 DEBUG [StoreOpener-25b84f3651d3c0adac7c0275b4cb1e0c-1] regionserver.HStore(539): loaded hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/25b84f3651d3c0adac7c0275b4cb1e0c/cf2/f70a7ebc6c464a08ab00a25e137bbc72.c09b5f7c44d4a83602090cbccc082b61->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf2/f70a7ebc6c464a08ab00a25e137bbc72-bottom 2023-04-12 02:55:42,872 DEBUG [StoreOpener-25b84f3651d3c0adac7c0275b4cb1e0c-1] regionserver.HStore(539): loaded hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/25b84f3651d3c0adac7c0275b4cb1e0c/cf2/fb8ce8dc74bd47fabd120f6f688962d5.c09b5f7c44d4a83602090cbccc082b61->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf2/fb8ce8dc74bd47fabd120f6f688962d5-bottom 2023-04-12 02:55:42,875 DEBUG [StoreOpener-25b84f3651d3c0adac7c0275b4cb1e0c-1] regionserver.HStore(539): loaded hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/25b84f3651d3c0adac7c0275b4cb1e0c/cf2/fe343f29642942069fb8e38396f28358.c09b5f7c44d4a83602090cbccc082b61->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf2/fe343f29642942069fb8e38396f28358-bottom 2023-04-12 02:55:42,876 INFO [StoreOpener-25b84f3651d3c0adac7c0275b4cb1e0c-1] regionserver.HStore(310): Store=25b84f3651d3c0adac7c0275b4cb1e0c/cf2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-12 02:55:42,876 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/25b84f3651d3c0adac7c0275b4cb1e0c 2023-04-12 02:55:42,877 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/25b84f3651d3c0adac7c0275b4cb1e0c 2023-04-12 02:55:42,879 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testRecoveryAndDoubleExecution descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-04-12 02:55:42,880 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 25b84f3651d3c0adac7c0275b4cb1e0c 2023-04-12 02:55:42,880 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 25b84f3651d3c0adac7c0275b4cb1e0c; next sequenceid=86; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=68846198, jitterRate=0.025888293981552124}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-04-12 02:55:42,881 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 25b84f3651d3c0adac7c0275b4cb1e0c: 2023-04-12 02:55:42,881 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRecoveryAndDoubleExecution,,1681268138347.25b84f3651d3c0adac7c0275b4cb1e0c., pid=118, masterSystemTime=1681268142608 2023-04-12 02:55:42,881 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: Opening Region; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 02:55:42,882 DEBUG [RS:2;jenkins-hbase4:43091-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-12 02:55:42,882 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: Opening Region; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 02:55:42,882 DEBUG [RS:2;jenkins-hbase4:43091-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-12 02:55:42,882 INFO [RS:2;jenkins-hbase4:43091-shortCompactions-0] regionserver.HStore(1898): Keeping/Overriding Compaction request priority to -2147482648 for CF cf1 since it belongs to recently split daughter region testRecoveryAndDoubleExecution,,1681268138347.25b84f3651d3c0adac7c0275b4cb1e0c. 2023-04-12 02:55:42,882 DEBUG [RS:2;jenkins-hbase4:43091-shortCompactions-0] regionserver.HStore(1912): 25b84f3651d3c0adac7c0275b4cb1e0c/cf1 is initiating minor compaction (all files) 2023-04-12 02:55:42,882 INFO [RS:2;jenkins-hbase4:43091-longCompactions-0] regionserver.HStore(1898): Keeping/Overriding Compaction request priority to -2147482648 for CF cf2 since it belongs to recently split daughter region testRecoveryAndDoubleExecution,,1681268138347.25b84f3651d3c0adac7c0275b4cb1e0c. 2023-04-12 02:55:42,883 INFO [RS:2;jenkins-hbase4:43091-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 25b84f3651d3c0adac7c0275b4cb1e0c/cf1 in testRecoveryAndDoubleExecution,,1681268138347.25b84f3651d3c0adac7c0275b4cb1e0c. 2023-04-12 02:55:42,883 DEBUG [RS:2;jenkins-hbase4:43091-longCompactions-0] regionserver.HStore(1912): 25b84f3651d3c0adac7c0275b4cb1e0c/cf2 is initiating minor compaction (all files) 2023-04-12 02:55:42,883 INFO [RS:2;jenkins-hbase4:43091-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 25b84f3651d3c0adac7c0275b4cb1e0c/cf2 in testRecoveryAndDoubleExecution,,1681268138347.25b84f3651d3c0adac7c0275b4cb1e0c. 2023-04-12 02:55:42,883 INFO [RS:2;jenkins-hbase4:43091-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/25b84f3651d3c0adac7c0275b4cb1e0c/cf1/a79c7cc5c59c4f6bbe6a6a584d824f36.c09b5f7c44d4a83602090cbccc082b61->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf1/a79c7cc5c59c4f6bbe6a6a584d824f36-bottom, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/25b84f3651d3c0adac7c0275b4cb1e0c/cf1/63025bd542924f2caced3f22c416a5a1.c09b5f7c44d4a83602090cbccc082b61->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf1/63025bd542924f2caced3f22c416a5a1-bottom, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/25b84f3651d3c0adac7c0275b4cb1e0c/cf1/9d5ec6ededfd4de8b6d44e635d18346c.c09b5f7c44d4a83602090cbccc082b61->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf1/9d5ec6ededfd4de8b6d44e635d18346c-bottom, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/25b84f3651d3c0adac7c0275b4cb1e0c/cf1/001e134c60444ac2b388f9df6bf5670b.c09b5f7c44d4a83602090cbccc082b61->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf1/001e134c60444ac2b388f9df6bf5670b-bottom, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/25b84f3651d3c0adac7c0275b4cb1e0c/cf1/c14bd2fe66004d6d842c72f4b2a4ee90.c09b5f7c44d4a83602090cbccc082b61->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf1/c14bd2fe66004d6d842c72f4b2a4ee90-bottom, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/25b84f3651d3c0adac7c0275b4cb1e0c/cf1/1373dcc8adeb4ae4b9929ef46fbb4980.c09b5f7c44d4a83602090cbccc082b61->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf1/1373dcc8adeb4ae4b9929ef46fbb4980-bottom, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/25b84f3651d3c0adac7c0275b4cb1e0c/cf1/e62dda4fe11148e88206bdd397ca64c0.c09b5f7c44d4a83602090cbccc082b61->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf1/e62dda4fe11148e88206bdd397ca64c0-bottom] into tmpdir=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/25b84f3651d3c0adac7c0275b4cb1e0c/.tmp, totalSize=34.9 K 2023-04-12 02:55:42,883 INFO [RS:2;jenkins-hbase4:43091-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/25b84f3651d3c0adac7c0275b4cb1e0c/cf2/5b22fe6156a04c8c9486fd6f189bf4d7.c09b5f7c44d4a83602090cbccc082b61->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf2/5b22fe6156a04c8c9486fd6f189bf4d7-bottom, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/25b84f3651d3c0adac7c0275b4cb1e0c/cf2/a4e6a82328b24916838ccb970ddd7704.c09b5f7c44d4a83602090cbccc082b61->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf2/a4e6a82328b24916838ccb970ddd7704-bottom, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/25b84f3651d3c0adac7c0275b4cb1e0c/cf2/fb8ce8dc74bd47fabd120f6f688962d5.c09b5f7c44d4a83602090cbccc082b61->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf2/fb8ce8dc74bd47fabd120f6f688962d5-bottom, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/25b84f3651d3c0adac7c0275b4cb1e0c/cf2/fe343f29642942069fb8e38396f28358.c09b5f7c44d4a83602090cbccc082b61->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf2/fe343f29642942069fb8e38396f28358-bottom, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/25b84f3651d3c0adac7c0275b4cb1e0c/cf2/1b3fc837d85449cba8b914aa7f18f0fe.c09b5f7c44d4a83602090cbccc082b61->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf2/1b3fc837d85449cba8b914aa7f18f0fe-bottom, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/25b84f3651d3c0adac7c0275b4cb1e0c/cf2/a7d4784e93134ef2943643862fd08463.c09b5f7c44d4a83602090cbccc082b61->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf2/a7d4784e93134ef2943643862fd08463-bottom, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/25b84f3651d3c0adac7c0275b4cb1e0c/cf2/f70a7ebc6c464a08ab00a25e137bbc72.c09b5f7c44d4a83602090cbccc082b61->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf2/f70a7ebc6c464a08ab00a25e137bbc72-bottom] into tmpdir=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/25b84f3651d3c0adac7c0275b4cb1e0c/.tmp, totalSize=34.9 K 2023-04-12 02:55:42,884 DEBUG [RS:2;jenkins-hbase4:43091-longCompactions-0] compactions.Compactor(207): Compacting 5b22fe6156a04c8c9486fd6f189bf4d7.c09b5f7c44d4a83602090cbccc082b61, keycount=1, bloomtype=ROW, size=4.8 K, encoding=NONE, compression=NONE, seqNum=6, earliestPutTs=1681268136828 2023-04-12 02:55:42,884 DEBUG [RS:2;jenkins-hbase4:43091-shortCompactions-0] compactions.Compactor(207): Compacting a79c7cc5c59c4f6bbe6a6a584d824f36.c09b5f7c44d4a83602090cbccc082b61, keycount=1, bloomtype=ROW, size=4.8 K, encoding=NONE, compression=NONE, seqNum=6, earliestPutTs=1681268136828 2023-04-12 02:55:42,884 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRecoveryAndDoubleExecution,,1681268138347.25b84f3651d3c0adac7c0275b4cb1e0c. 2023-04-12 02:55:42,884 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRecoveryAndDoubleExecution,,1681268138347.25b84f3651d3c0adac7c0275b4cb1e0c. 2023-04-12 02:55:42,884 DEBUG [RS:2;jenkins-hbase4:43091-longCompactions-0] compactions.Compactor(207): Compacting a4e6a82328b24916838ccb970ddd7704.c09b5f7c44d4a83602090cbccc082b61, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=19, earliestPutTs=1681268137049 2023-04-12 02:55:42,885 DEBUG [RS:2;jenkins-hbase4:43091-shortCompactions-0] compactions.Compactor(207): Compacting 63025bd542924f2caced3f22c416a5a1.c09b5f7c44d4a83602090cbccc082b61, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=19, earliestPutTs=1681268137049 2023-04-12 02:55:42,885 DEBUG [RS:2;jenkins-hbase4:43091-longCompactions-0] compactions.Compactor(207): Compacting fb8ce8dc74bd47fabd120f6f688962d5.c09b5f7c44d4a83602090cbccc082b61, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=32, earliestPutTs=1681268137249 2023-04-12 02:55:42,885 DEBUG [RS:2;jenkins-hbase4:43091-shortCompactions-0] compactions.Compactor(207): Compacting 9d5ec6ededfd4de8b6d44e635d18346c.c09b5f7c44d4a83602090cbccc082b61, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=32, earliestPutTs=1681268137249 2023-04-12 02:55:42,886 DEBUG [RS:2;jenkins-hbase4:43091-longCompactions-0] compactions.Compactor(207): Compacting fe343f29642942069fb8e38396f28358.c09b5f7c44d4a83602090cbccc082b61, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=45, earliestPutTs=1681268137466 2023-04-12 02:55:42,886 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=115 updating hbase:meta row=25b84f3651d3c0adac7c0275b4cb1e0c, regionState=OPEN, openSeqNum=86, regionLocation=jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:42,886 DEBUG [RS:2;jenkins-hbase4:43091-shortCompactions-0] compactions.Compactor(207): Compacting 001e134c60444ac2b388f9df6bf5670b.c09b5f7c44d4a83602090cbccc082b61, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=45, earliestPutTs=1681268137466 2023-04-12 02:55:42,886 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRecoveryAndDoubleExecution,,1681268138347.25b84f3651d3c0adac7c0275b4cb1e0c.","families":{"info":[{"qualifier":"regioninfo","vlen":66,"tag":[],"timestamp":"1681268142885"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1681268142885"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681268142885"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681268142885"}]},"ts":"1681268142885"} 2023-04-12 02:55:42,886 DEBUG [RS:2;jenkins-hbase4:43091-longCompactions-0] compactions.Compactor(207): Compacting 1b3fc837d85449cba8b914aa7f18f0fe.c09b5f7c44d4a83602090cbccc082b61, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=58, earliestPutTs=1681268137658 2023-04-12 02:55:42,886 DEBUG [RS:2;jenkins-hbase4:43091-shortCompactions-0] compactions.Compactor(207): Compacting c14bd2fe66004d6d842c72f4b2a4ee90.c09b5f7c44d4a83602090cbccc082b61, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=58, earliestPutTs=1681268137658 2023-04-12 02:55:42,887 DEBUG [RS:2;jenkins-hbase4:43091-longCompactions-0] compactions.Compactor(207): Compacting a7d4784e93134ef2943643862fd08463.c09b5f7c44d4a83602090cbccc082b61, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=71, earliestPutTs=1681268137880 2023-04-12 02:55:42,888 DEBUG [RS:2;jenkins-hbase4:43091-shortCompactions-0] compactions.Compactor(207): Compacting 1373dcc8adeb4ae4b9929ef46fbb4980.c09b5f7c44d4a83602090cbccc082b61, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=71, earliestPutTs=1681268137880 2023-04-12 02:55:42,888 DEBUG [RS:2;jenkins-hbase4:43091-longCompactions-0] compactions.Compactor(207): Compacting f70a7ebc6c464a08ab00a25e137bbc72.c09b5f7c44d4a83602090cbccc082b61, keycount=4, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=82, earliestPutTs=1681268138088 2023-04-12 02:55:42,890 DEBUG [RS:2;jenkins-hbase4:43091-shortCompactions-0] compactions.Compactor(207): Compacting e62dda4fe11148e88206bdd397ca64c0.c09b5f7c44d4a83602090cbccc082b61, keycount=4, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=82, earliestPutTs=1681268138088 2023-04-12 02:55:42,896 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=118, resume processing ppid=115 2023-04-12 02:55:42,896 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=118, ppid=115, state=SUCCESS; OpenRegionProcedure 25b84f3651d3c0adac7c0275b4cb1e0c, server=jenkins-hbase4.apache.org,43091,1681268031804 in 434 msec 2023-04-12 02:55:42,899 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=115, resume processing ppid=112 2023-04-12 02:55:42,899 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=115, ppid=112, state=SUCCESS; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=25b84f3651d3c0adac7c0275b4cb1e0c, ASSIGN in 596 msec 2023-04-12 02:55:42,899 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-04-12 02:55:42,899 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=112, state=SUCCESS, locked=true; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=c09b5f7c44d4a83602090cbccc082b61, daughterA=25b84f3651d3c0adac7c0275b4cb1e0c, daughterB=205d5359b7ecb778c3d784ef1bb72d34 2023-04-12 02:55:42,899 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-04-12 02:55:42,899 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=112, state=SUCCESS, locked=true; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=c09b5f7c44d4a83602090cbccc082b61, daughterA=25b84f3651d3c0adac7c0275b4cb1e0c, daughterB=205d5359b7ecb778c3d784ef1bb72d34 at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-04-12 02:55:42,904 INFO [RS:2;jenkins-hbase4:43091-longCompactions-0] throttle.PressureAwareThroughputController(145): 25b84f3651d3c0adac7c0275b4cb1e0c#cf2#compaction#110 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 02:55:42,907 INFO [RS:2;jenkins-hbase4:43091-shortCompactions-0] throttle.PressureAwareThroughputController(145): 25b84f3651d3c0adac7c0275b4cb1e0c#cf1#compaction#111 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 02:55:42,943 DEBUG [RS:2;jenkins-hbase4:43091-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/25b84f3651d3c0adac7c0275b4cb1e0c/.tmp/cf2/8de8d9f320ee4e34aac7d6d6450e1472 as hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/25b84f3651d3c0adac7c0275b4cb1e0c/cf2/8de8d9f320ee4e34aac7d6d6450e1472 2023-04-12 02:55:42,951 DEBUG [RS:2;jenkins-hbase4:43091-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/25b84f3651d3c0adac7c0275b4cb1e0c/.tmp/cf1/8bb8c95221324ace979de6f9e8ee711c as hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/25b84f3651d3c0adac7c0275b4cb1e0c/cf1/8bb8c95221324ace979de6f9e8ee711c 2023-04-12 02:55:42,952 INFO [RS:2;jenkins-hbase4:43091-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 25b84f3651d3c0adac7c0275b4cb1e0c/cf2 of 25b84f3651d3c0adac7c0275b4cb1e0c into 8de8d9f320ee4e34aac7d6d6450e1472(size=6.1 K), total size for store is 6.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 02:55:42,952 DEBUG [RS:2;jenkins-hbase4:43091-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 25b84f3651d3c0adac7c0275b4cb1e0c: 2023-04-12 02:55:42,952 INFO [RS:2;jenkins-hbase4:43091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=testRecoveryAndDoubleExecution,,1681268138347.25b84f3651d3c0adac7c0275b4cb1e0c., storeName=25b84f3651d3c0adac7c0275b4cb1e0c/cf2, priority=9, startTime=1681268142882; duration=0sec 2023-04-12 02:55:42,953 DEBUG [RS:2;jenkins-hbase4:43091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 02:55:42,957 INFO [RS:2;jenkins-hbase4:43091-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 25b84f3651d3c0adac7c0275b4cb1e0c/cf1 of 25b84f3651d3c0adac7c0275b4cb1e0c into 8bb8c95221324ace979de6f9e8ee711c(size=6.1 K), total size for store is 6.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 02:55:42,957 DEBUG [RS:2;jenkins-hbase4:43091-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 25b84f3651d3c0adac7c0275b4cb1e0c: 2023-04-12 02:55:42,957 INFO [RS:2;jenkins-hbase4:43091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=testRecoveryAndDoubleExecution,,1681268138347.25b84f3651d3c0adac7c0275b4cb1e0c., storeName=25b84f3651d3c0adac7c0275b4cb1e0c/cf1, priority=9, startTime=1681268142881; duration=0sec 2023-04-12 02:55:42,957 DEBUG [RS:2;jenkins-hbase4:43091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 02:55:43,050 INFO [Listener at localhost/46667] procedure.MasterProcedureTestingUtility(454): Restart 10 exec state=pid=112, state=SUCCESS, locked=true; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=c09b5f7c44d4a83602090cbccc082b61, daughterA=25b84f3651d3c0adac7c0275b4cb1e0c, daughterB=205d5359b7ecb778c3d784ef1bb72d34 2023-04-12 02:55:43,051 INFO [Listener at localhost/46667] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-04-12 02:55:43,051 INFO [Listener at localhost/46667] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-04-12 02:55:43,051 INFO [Listener at localhost/46667] assignment.AssignmentManager(315): Stopping assignment manager 2023-04-12 02:55:43,051 ERROR [Listener at localhost/46667] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-04-12 02:55:43,051 INFO [Listener at localhost/46667] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-04-12 02:55:43,051 INFO [Listener at localhost/46667] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-04-12 02:55:43,051 INFO [Listener at localhost/46667] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-04-12 02:55:43,052 INFO [Listener at localhost/46667] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-04-12 02:55:43,052 INFO [Listener at localhost/46667] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-04-12 02:55:43,054 DEBUG [Listener at localhost/46667] store.ProcedureTree(114): Procedure Procedure(pid=112, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[0, 1, 2, 7, 8, 9, 10, 11, 12, 13] 2023-04-12 02:55:43,054 DEBUG [Listener at localhost/46667] store.ProcedureTree(114): Procedure Procedure(pid=113, ppid=112, class=org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure) stack ids=[3, 6] 2023-04-12 02:55:43,054 DEBUG [Listener at localhost/46667] store.ProcedureTree(114): Procedure Procedure(pid=114, ppid=113, class=org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure) stack ids=[4, 5] 2023-04-12 02:55:43,054 DEBUG [Listener at localhost/46667] store.ProcedureTree(114): Procedure Procedure(pid=115, ppid=112, class=org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure) stack ids=[15, 18, 23] 2023-04-12 02:55:43,054 DEBUG [Listener at localhost/46667] store.ProcedureTree(114): Procedure Procedure(pid=118, ppid=115, class=org.apache.hadoop.hbase.master.assignment.OpenRegionProcedure) stack ids=[19, 22] 2023-04-12 02:55:43,054 DEBUG [Listener at localhost/46667] store.ProcedureTree(114): Procedure Procedure(pid=116, ppid=112, class=org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure) stack ids=[14, 16, 21] 2023-04-12 02:55:43,054 DEBUG [Listener at localhost/46667] store.ProcedureTree(114): Procedure Procedure(pid=117, ppid=116, class=org.apache.hadoop.hbase.master.assignment.OpenRegionProcedure) stack ids=[17, 20] 2023-04-12 02:55:43,055 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-04-12 02:55:43,055 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-04-12 02:55:43,055 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-04-12 02:55:43,056 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-04-12 02:55:43,056 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-12 02:55:43,056 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=cc9012ec0b656baa24d04368fbe026c4, daughterA=e2a9b8fe15a781798c0b49e48ff5aad7, daughterB=6f55b5d6a460bd8c7569dee22f049104 2023-04-12 02:55:43,056 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-12 02:55:43,056 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-12 02:55:43,057 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-04-12 02:55:43,057 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=28, state=SUCCESS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=315e13e868a7f68a3ce37802b2f6312d, daughterA=4e1092f3c00484f8f09c730861225d43, daughterB=291734c00d212fd203b04cf8872d4649 2023-04-12 02:55:43,057 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=35, state=SUCCESS; DisableTableProcedure table=testSplitWithoutPONR 2023-04-12 02:55:43,057 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=40, state=SUCCESS; DeleteTableProcedure table=testSplitWithoutPONR 2023-04-12 02:55:43,057 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=41, state=SUCCESS; CreateTableProcedure table=testRollbackAndDoubleExecution 2023-04-12 02:55:43,058 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=44, state=ROLLEDBACK, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=96cdae2f7638212730b694522b9a464a, daughterA=fdd18e9cf539c13d863267ce16031583, daughterB=578095173fbc35e181cce14464037c2e 2023-04-12 02:55:43,058 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=47, state=SUCCESS; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=96cdae2f7638212730b694522b9a464a, ASSIGN 2023-04-12 02:55:43,058 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=49, state=SUCCESS; DisableTableProcedure table=testRollbackAndDoubleExecution 2023-04-12 02:55:43,058 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=52, state=SUCCESS; DeleteTableProcedure table=testRollbackAndDoubleExecution 2023-04-12 02:55:43,058 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=53, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionNoStoreFile 2023-04-12 02:55:43,059 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=56, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionNoStoreFile, parent=947433d660677a65ad9876ce2025e3e0, daughterA=b24cea6994eaa39f3128573197d02d9b, daughterB=b0e3d5fd7da33cb33084791956f0d7ca 2023-04-12 02:55:43,059 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=63, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionNoStoreFile 2023-04-12 02:55:43,059 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=68, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionNoStoreFile 2023-04-12 02:55:43,059 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=69, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:55:43,059 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=72, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionDeletedRowsDaughter, parent=1c6b8f18205322cbb248ea4fb10155d8, daughterA=81296e35c0d27ffa1ba7676ff53f87e1, daughterB=72b78dfd0678a920ddc3700612ff3932 2023-04-12 02:55:43,059 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=79, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:55:43,059 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=84, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-04-12 02:55:43,060 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=85, state=SUCCESS; CreateTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-04-12 02:55:43,060 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=88, state=SUCCESS; ModifyTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-04-12 02:55:43,060 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=95, state=ROLLEDBACK, exception=org.apache.hadoop.hbase.HBaseIOException via master-split-regions:org.apache.hadoop.hbase.HBaseIOException: The parent region state=OPENING, location=jenkins-hbase4.apache.org,34579,1681268031719, table=testRollbackForSplitTableRegionWithReplica, region=94855f8e18950d1c0c1e1d85df03f9ad is currently in transition, give up; SplitTableRegionProcedure table=testRollbackForSplitTableRegionWithReplica, parent=4381c9c8e57c5dbb2533908abaf07cf1, daughterA=2e35c2f6b8767f438f484c700a605299, daughterB=8407ddb6fc7704d121cd7f8a674aefb6 2023-04-12 02:55:43,060 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=96, state=SUCCESS; DisableTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-04-12 02:55:43,061 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=101, state=SUCCESS; DeleteTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-04-12 02:55:43,061 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=102, state=SUCCESS; CreateTableProcedure table=testInvalidSplitKey 2023-04-12 02:55:43,061 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=105, state=SUCCESS; DisableTableProcedure table=testInvalidSplitKey 2023-04-12 02:55:43,061 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=108, state=SUCCESS; DeleteTableProcedure table=testInvalidSplitKey 2023-04-12 02:55:43,061 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(411): Completed pid=109, state=SUCCESS; CreateTableProcedure table=testRecoveryAndDoubleExecution 2023-04-12 02:55:43,063 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(468): Loading pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_POST_OPERATION; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=c09b5f7c44d4a83602090cbccc082b61, daughterA=25b84f3651d3c0adac7c0275b4cb1e0c, daughterB=205d5359b7ecb778c3d784ef1bb72d34 2023-04-12 02:55:43,063 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(468): Loading pid=113, ppid=112, state=SUCCESS; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=c09b5f7c44d4a83602090cbccc082b61, UNASSIGN 2023-04-12 02:55:43,063 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(468): Loading pid=114, ppid=113, state=SUCCESS; CloseRegionProcedure c09b5f7c44d4a83602090cbccc082b61, server=jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:43,063 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(468): Loading pid=115, ppid=112, state=SUCCESS; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=25b84f3651d3c0adac7c0275b4cb1e0c, ASSIGN 2023-04-12 02:55:43,063 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(468): Loading pid=116, ppid=112, state=SUCCESS; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=205d5359b7ecb778c3d784ef1bb72d34, ASSIGN 2023-04-12 02:55:43,063 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(468): Loading pid=117, ppid=116, state=SUCCESS; OpenRegionProcedure 205d5359b7ecb778c3d784ef1bb72d34, server=jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:43,063 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(468): Loading pid=118, ppid=115, state=SUCCESS; OpenRegionProcedure 25b84f3651d3c0adac7c0275b4cb1e0c, server=jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:43,063 DEBUG [Listener at localhost/46667] procedure2.Procedure(966): pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_POST_OPERATION; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=c09b5f7c44d4a83602090cbccc082b61, daughterA=25b84f3651d3c0adac7c0275b4cb1e0c, daughterB=205d5359b7ecb778c3d784ef1bb72d34 held the lock before restarting, call acquireLock to restore it. 2023-04-12 02:55:43,063 INFO [Listener at localhost/46667] procedure.MasterProcedureScheduler(727): Took xlock for pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_POST_OPERATION; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=c09b5f7c44d4a83602090cbccc082b61, daughterA=25b84f3651d3c0adac7c0275b4cb1e0c, daughterB=205d5359b7ecb778c3d784ef1bb72d34 2023-04-12 02:55:43,064 INFO [Listener at localhost/46667] procedure.MasterProcedureScheduler(727): Took xlock for pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_POST_OPERATION; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=c09b5f7c44d4a83602090cbccc082b61, daughterA=25b84f3651d3c0adac7c0275b4cb1e0c, daughterB=205d5359b7ecb778c3d784ef1bb72d34 2023-04-12 02:55:43,064 INFO [Listener at localhost/46667] procedure.MasterProcedureScheduler(727): Took xlock for pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_POST_OPERATION; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=c09b5f7c44d4a83602090cbccc082b61, daughterA=25b84f3651d3c0adac7c0275b4cb1e0c, daughterB=205d5359b7ecb778c3d784ef1bb72d34 2023-04-12 02:55:43,064 DEBUG [Listener at localhost/46667] procedure2.Procedure(946): pid=113, ppid=112, state=SUCCESS; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=c09b5f7c44d4a83602090cbccc082b61, UNASSIGN didn't hold the lock before restarting, skip acquiring lock. 2023-04-12 02:55:43,064 DEBUG [Listener at localhost/46667] procedure2.Procedure(946): pid=114, ppid=113, state=SUCCESS; CloseRegionProcedure c09b5f7c44d4a83602090cbccc082b61, server=jenkins-hbase4.apache.org,43091,1681268031804 didn't hold the lock before restarting, skip acquiring lock. 2023-04-12 02:55:43,064 DEBUG [Listener at localhost/46667] procedure2.Procedure(946): pid=115, ppid=112, state=SUCCESS; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=25b84f3651d3c0adac7c0275b4cb1e0c, ASSIGN didn't hold the lock before restarting, skip acquiring lock. 2023-04-12 02:55:43,064 DEBUG [Listener at localhost/46667] procedure2.Procedure(946): pid=116, ppid=112, state=SUCCESS; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=205d5359b7ecb778c3d784ef1bb72d34, ASSIGN didn't hold the lock before restarting, skip acquiring lock. 2023-04-12 02:55:43,064 DEBUG [Listener at localhost/46667] procedure2.Procedure(946): pid=117, ppid=116, state=SUCCESS; OpenRegionProcedure 205d5359b7ecb778c3d784ef1bb72d34, server=jenkins-hbase4.apache.org,43091,1681268031804 didn't hold the lock before restarting, skip acquiring lock. 2023-04-12 02:55:43,064 DEBUG [Listener at localhost/46667] procedure2.Procedure(946): pid=118, ppid=115, state=SUCCESS; OpenRegionProcedure 25b84f3651d3c0adac7c0275b4cb1e0c, server=jenkins-hbase4.apache.org,43091,1681268031804 didn't hold the lock before restarting, skip acquiring lock. 2023-04-12 02:55:43,064 INFO [Listener at localhost/46667] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 11 msec 2023-04-12 02:55:43,065 DEBUG [Listener at localhost/46667] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-04-12 02:55:43,065 DEBUG [Listener at localhost/46667] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase4.apache.org,44679,1681268031772, table=hbase:meta, region=1588230740 2023-04-12 02:55:43,067 INFO [Listener at localhost/46667] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1681268173067 2023-04-12 02:55:43,069 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_POST_OPERATION; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=c09b5f7c44d4a83602090cbccc082b61, daughterA=25b84f3651d3c0adac7c0275b4cb1e0c, daughterB=205d5359b7ecb778c3d784ef1bb72d34 2023-04-12 02:55:43,069 DEBUG [Listener at localhost/46667] assignment.AssignmentManager(1517): Joining cluster... 2023-04-12 02:55:43,072 DEBUG [Listener at localhost/46667] assignment.RegionStateStore(147): Load hbase:meta entry region=1f255e980a31bb266f2206a30aa2ae3f, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,44679,1681268031772, regionLocation=jenkins-hbase4.apache.org,44679,1681268031772, openSeqNum=2 2023-04-12 02:55:43,073 DEBUG [Listener at localhost/46667] assignment.RegionStateStore(147): Load hbase:meta entry region=c09b5f7c44d4a83602090cbccc082b61, regionState=CLOSED, lastHost=jenkins-hbase4.apache.org,43091,1681268031804, regionLocation=jenkins-hbase4.apache.org,43091,1681268031804, openSeqNum=2 2023-04-12 02:55:43,073 INFO [Listener at localhost/46667] assignment.RegionStates(657): Added to offline, CURRENTLY NEVER CLEARED!!! state=CLOSED, location=jenkins-hbase4.apache.org,43091,1681268031804, table=testRecoveryAndDoubleExecution, region=c09b5f7c44d4a83602090cbccc082b61 2023-04-12 02:55:43,073 DEBUG [Listener at localhost/46667] assignment.RegionStateStore(147): Load hbase:meta entry region=25b84f3651d3c0adac7c0275b4cb1e0c, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,43091,1681268031804, regionLocation=jenkins-hbase4.apache.org,43091,1681268031804, openSeqNum=86 2023-04-12 02:55:43,073 DEBUG [Listener at localhost/46667] assignment.RegionStateStore(147): Load hbase:meta entry region=205d5359b7ecb778c3d784ef1bb72d34, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,43091,1681268031804, regionLocation=jenkins-hbase4.apache.org,43091,1681268031804, openSeqNum=86 2023-04-12 02:55:43,073 INFO [Listener at localhost/46667] assignment.AssignmentManager(1529): Number of RegionServers=3 2023-04-12 02:55:43,073 INFO [Listener at localhost/46667] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1681268203073 2023-04-12 02:55:43,074 INFO [Listener at localhost/46667] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1681268263074 2023-04-12 02:55:43,074 INFO [Listener at localhost/46667] assignment.AssignmentManager(1536): Joined the cluster in 4 msec 2023-04-12 02:55:43,074 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-04-12 02:55:43,084 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=112, state=SUCCESS; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=c09b5f7c44d4a83602090cbccc082b61, daughterA=25b84f3651d3c0adac7c0275b4cb1e0c, daughterB=205d5359b7ecb778c3d784ef1bb72d34 in 4.7270 sec 2023-04-12 02:55:43,300 ERROR [BP-1247239137-172.31.14.131-1681268026808 heartbeating to localhost/127.0.0.1:44691] datanode.BPServiceActor(848): Exception in BPOfferService for Block pool BP-1247239137-172.31.14.131-1681268026808 (Datanode Uuid 569a15e2-91b7-43b5-bb2b-d70e35911578) service to localhost/127.0.0.1:44691 java.lang.IllegalThreadStateException at java.lang.ThreadGroup.addUnstarted(ThreadGroup.java:867) at java.lang.Thread.init(Thread.java:407) at java.lang.Thread.init(Thread.java:351) at java.lang.Thread.(Thread.java:501) at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService$1.newThread(FsDatasetAsyncDiskService.java:102) at java.util.concurrent.ThreadPoolExecutor$Worker.(ThreadPoolExecutor.java:619) at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:932) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1367) at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService.execute(FsDatasetAsyncDiskService.java:172) at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService.deleteAsync(FsDatasetAsyncDiskService.java:221) at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.invalidate(FsDatasetImpl.java:2216) at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.invalidate(FsDatasetImpl.java:2140) at org.apache.hadoop.hdfs.server.datanode.BPOfferService.processCommandFromActive(BPOfferService.java:732) at org.apache.hadoop.hdfs.server.datanode.BPOfferService.processCommandFromActor(BPOfferService.java:679) at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.processCommand(BPServiceActor.java:880) at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:676) at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:846) at java.lang.Thread.run(Thread.java:750) 2023-04-12 02:55:43,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43091] ipc.CallRunner(144): callId: 872 service: ClientService methodName: Scan size: 154 connection: 172.31.14.131:38048 deadline: 1681268203325, exception=org.apache.hadoop.hbase.NotServingRegionException: testRecoveryAndDoubleExecution,,1681268136320.c09b5f7c44d4a83602090cbccc082b61. is not online on jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:43,436 DEBUG [Listener at localhost/46667] client.ClientScanner(240): Advancing internal scanner to startKey at '41', inclusive 2023-04-12 02:55:43,443 INFO [Listener at localhost/46667] assignment.TestSplitTableRegionProcedure(570): Row Count = 60 2023-04-12 02:55:43,444 DEBUG [Listener at localhost/46667] client.ClientScanner(240): Advancing internal scanner to startKey at '41', inclusive 2023-04-12 02:55:43,454 WARN [Listener at localhost/46667] procedure2.ProcedureTestingUtility(193): Set Kill before store update to: false 2023-04-12 02:55:43,456 INFO [Listener at localhost/46667] client.HBaseAdmin$15(890): Started disable of testRecoveryAndDoubleExecution 2023-04-12 02:55:43,456 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable testRecoveryAndDoubleExecution 2023-04-12 02:55:43,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] procedure2.ProcedureExecutor(1029): Stored pid=119, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testRecoveryAndDoubleExecution 2023-04-12 02:55:43,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] master.MasterRpcServices(1227): Checking to see if procedure is done pid=119 2023-04-12 02:55:43,463 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRecoveryAndDoubleExecution","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681268143463"}]},"ts":"1681268143463"} 2023-04-12 02:55:43,464 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testRecoveryAndDoubleExecution, state=DISABLING in hbase:meta 2023-04-12 02:55:43,465 INFO [PEWorker-1] procedure.DisableTableProcedure(293): Set testRecoveryAndDoubleExecution to state=DISABLING 2023-04-12 02:55:43,466 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=120, ppid=119, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=25b84f3651d3c0adac7c0275b4cb1e0c, UNASSIGN}, {pid=121, ppid=119, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=205d5359b7ecb778c3d784ef1bb72d34, UNASSIGN}] 2023-04-12 02:55:43,466 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=121, ppid=119, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=205d5359b7ecb778c3d784ef1bb72d34, UNASSIGN 2023-04-12 02:55:43,467 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=121 updating hbase:meta row=205d5359b7ecb778c3d784ef1bb72d34, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:43,467 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRecoveryAndDoubleExecution,41,1681268138347.205d5359b7ecb778c3d784ef1bb72d34.","families":{"info":[{"qualifier":"regioninfo","vlen":66,"tag":[],"timestamp":"1681268143467"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681268143467"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681268143467"}]},"ts":"1681268143467"} 2023-04-12 02:55:43,468 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=122, ppid=121, state=RUNNABLE; CloseRegionProcedure 205d5359b7ecb778c3d784ef1bb72d34, server=jenkins-hbase4.apache.org,43091,1681268031804}] 2023-04-12 02:55:43,469 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=120, ppid=119, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=25b84f3651d3c0adac7c0275b4cb1e0c, UNASSIGN 2023-04-12 02:55:43,470 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=120 updating hbase:meta row=25b84f3651d3c0adac7c0275b4cb1e0c, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:43,470 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRecoveryAndDoubleExecution,,1681268138347.25b84f3651d3c0adac7c0275b4cb1e0c.","families":{"info":[{"qualifier":"regioninfo","vlen":66,"tag":[],"timestamp":"1681268143470"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681268143470"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681268143470"}]},"ts":"1681268143470"} 2023-04-12 02:55:43,471 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=123, ppid=120, state=RUNNABLE; CloseRegionProcedure 25b84f3651d3c0adac7c0275b4cb1e0c, server=jenkins-hbase4.apache.org,43091,1681268031804}] 2023-04-12 02:55:43,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] master.MasterRpcServices(1227): Checking to see if procedure is done pid=119 2023-04-12 02:55:43,619 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 205d5359b7ecb778c3d784ef1bb72d34 2023-04-12 02:55:43,620 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 205d5359b7ecb778c3d784ef1bb72d34, disabling compactions & flushes 2023-04-12 02:55:43,620 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region testRecoveryAndDoubleExecution,41,1681268138347.205d5359b7ecb778c3d784ef1bb72d34. 2023-04-12 02:55:43,620 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testRecoveryAndDoubleExecution,41,1681268138347.205d5359b7ecb778c3d784ef1bb72d34. 2023-04-12 02:55:43,620 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on testRecoveryAndDoubleExecution,41,1681268138347.205d5359b7ecb778c3d784ef1bb72d34. after waiting 0 ms 2023-04-12 02:55:43,620 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region testRecoveryAndDoubleExecution,41,1681268138347.205d5359b7ecb778c3d784ef1bb72d34. 2023-04-12 02:55:43,620 DEBUG [StoreCloser-testRecoveryAndDoubleExecution,41,1681268138347.205d5359b7ecb778c3d784ef1bb72d34.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/205d5359b7ecb778c3d784ef1bb72d34/cf1/a79c7cc5c59c4f6bbe6a6a584d824f36.c09b5f7c44d4a83602090cbccc082b61->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf1/a79c7cc5c59c4f6bbe6a6a584d824f36-top, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/205d5359b7ecb778c3d784ef1bb72d34/cf1/63025bd542924f2caced3f22c416a5a1.c09b5f7c44d4a83602090cbccc082b61->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf1/63025bd542924f2caced3f22c416a5a1-top, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/205d5359b7ecb778c3d784ef1bb72d34/cf1/9d5ec6ededfd4de8b6d44e635d18346c.c09b5f7c44d4a83602090cbccc082b61->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf1/9d5ec6ededfd4de8b6d44e635d18346c-top, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/205d5359b7ecb778c3d784ef1bb72d34/cf1/001e134c60444ac2b388f9df6bf5670b.c09b5f7c44d4a83602090cbccc082b61->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf1/001e134c60444ac2b388f9df6bf5670b-top, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/205d5359b7ecb778c3d784ef1bb72d34/cf1/c14bd2fe66004d6d842c72f4b2a4ee90.c09b5f7c44d4a83602090cbccc082b61->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf1/c14bd2fe66004d6d842c72f4b2a4ee90-top, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/205d5359b7ecb778c3d784ef1bb72d34/cf1/1373dcc8adeb4ae4b9929ef46fbb4980.c09b5f7c44d4a83602090cbccc082b61->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf1/1373dcc8adeb4ae4b9929ef46fbb4980-top, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/205d5359b7ecb778c3d784ef1bb72d34/cf1/e62dda4fe11148e88206bdd397ca64c0.c09b5f7c44d4a83602090cbccc082b61->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf1/e62dda4fe11148e88206bdd397ca64c0-top] to archive 2023-04-12 02:55:43,621 DEBUG [StoreCloser-testRecoveryAndDoubleExecution,41,1681268138347.205d5359b7ecb778c3d784ef1bb72d34.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-12 02:55:43,623 DEBUG [StoreCloser-testRecoveryAndDoubleExecution,41,1681268138347.205d5359b7ecb778c3d784ef1bb72d34.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/205d5359b7ecb778c3d784ef1bb72d34/cf1/a79c7cc5c59c4f6bbe6a6a584d824f36.c09b5f7c44d4a83602090cbccc082b61 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testRecoveryAndDoubleExecution/205d5359b7ecb778c3d784ef1bb72d34/cf1/a79c7cc5c59c4f6bbe6a6a584d824f36.c09b5f7c44d4a83602090cbccc082b61 2023-04-12 02:55:43,624 DEBUG [StoreCloser-testRecoveryAndDoubleExecution,41,1681268138347.205d5359b7ecb778c3d784ef1bb72d34.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/205d5359b7ecb778c3d784ef1bb72d34/cf1/63025bd542924f2caced3f22c416a5a1.c09b5f7c44d4a83602090cbccc082b61 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testRecoveryAndDoubleExecution/205d5359b7ecb778c3d784ef1bb72d34/cf1/63025bd542924f2caced3f22c416a5a1.c09b5f7c44d4a83602090cbccc082b61 2023-04-12 02:55:43,625 DEBUG [StoreCloser-testRecoveryAndDoubleExecution,41,1681268138347.205d5359b7ecb778c3d784ef1bb72d34.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/205d5359b7ecb778c3d784ef1bb72d34/cf1/9d5ec6ededfd4de8b6d44e635d18346c.c09b5f7c44d4a83602090cbccc082b61 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testRecoveryAndDoubleExecution/205d5359b7ecb778c3d784ef1bb72d34/cf1/9d5ec6ededfd4de8b6d44e635d18346c.c09b5f7c44d4a83602090cbccc082b61 2023-04-12 02:55:43,627 DEBUG [StoreCloser-testRecoveryAndDoubleExecution,41,1681268138347.205d5359b7ecb778c3d784ef1bb72d34.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/205d5359b7ecb778c3d784ef1bb72d34/cf1/001e134c60444ac2b388f9df6bf5670b.c09b5f7c44d4a83602090cbccc082b61 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testRecoveryAndDoubleExecution/205d5359b7ecb778c3d784ef1bb72d34/cf1/001e134c60444ac2b388f9df6bf5670b.c09b5f7c44d4a83602090cbccc082b61 2023-04-12 02:55:43,628 DEBUG [StoreCloser-testRecoveryAndDoubleExecution,41,1681268138347.205d5359b7ecb778c3d784ef1bb72d34.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/205d5359b7ecb778c3d784ef1bb72d34/cf1/c14bd2fe66004d6d842c72f4b2a4ee90.c09b5f7c44d4a83602090cbccc082b61 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testRecoveryAndDoubleExecution/205d5359b7ecb778c3d784ef1bb72d34/cf1/c14bd2fe66004d6d842c72f4b2a4ee90.c09b5f7c44d4a83602090cbccc082b61 2023-04-12 02:55:43,629 DEBUG [StoreCloser-testRecoveryAndDoubleExecution,41,1681268138347.205d5359b7ecb778c3d784ef1bb72d34.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/205d5359b7ecb778c3d784ef1bb72d34/cf1/1373dcc8adeb4ae4b9929ef46fbb4980.c09b5f7c44d4a83602090cbccc082b61 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testRecoveryAndDoubleExecution/205d5359b7ecb778c3d784ef1bb72d34/cf1/1373dcc8adeb4ae4b9929ef46fbb4980.c09b5f7c44d4a83602090cbccc082b61 2023-04-12 02:55:43,634 DEBUG [StoreCloser-testRecoveryAndDoubleExecution,41,1681268138347.205d5359b7ecb778c3d784ef1bb72d34.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/205d5359b7ecb778c3d784ef1bb72d34/cf1/e62dda4fe11148e88206bdd397ca64c0.c09b5f7c44d4a83602090cbccc082b61 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testRecoveryAndDoubleExecution/205d5359b7ecb778c3d784ef1bb72d34/cf1/e62dda4fe11148e88206bdd397ca64c0.c09b5f7c44d4a83602090cbccc082b61 2023-04-12 02:55:43,635 DEBUG [StoreCloser-testRecoveryAndDoubleExecution,41,1681268138347.205d5359b7ecb778c3d784ef1bb72d34.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/205d5359b7ecb778c3d784ef1bb72d34/cf2/5b22fe6156a04c8c9486fd6f189bf4d7.c09b5f7c44d4a83602090cbccc082b61->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf2/5b22fe6156a04c8c9486fd6f189bf4d7-top, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/205d5359b7ecb778c3d784ef1bb72d34/cf2/a4e6a82328b24916838ccb970ddd7704.c09b5f7c44d4a83602090cbccc082b61->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf2/a4e6a82328b24916838ccb970ddd7704-top, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/205d5359b7ecb778c3d784ef1bb72d34/cf2/fb8ce8dc74bd47fabd120f6f688962d5.c09b5f7c44d4a83602090cbccc082b61->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf2/fb8ce8dc74bd47fabd120f6f688962d5-top, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/205d5359b7ecb778c3d784ef1bb72d34/cf2/fe343f29642942069fb8e38396f28358.c09b5f7c44d4a83602090cbccc082b61->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf2/fe343f29642942069fb8e38396f28358-top, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/205d5359b7ecb778c3d784ef1bb72d34/cf2/1b3fc837d85449cba8b914aa7f18f0fe.c09b5f7c44d4a83602090cbccc082b61->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf2/1b3fc837d85449cba8b914aa7f18f0fe-top, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/205d5359b7ecb778c3d784ef1bb72d34/cf2/a7d4784e93134ef2943643862fd08463.c09b5f7c44d4a83602090cbccc082b61->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf2/a7d4784e93134ef2943643862fd08463-top, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/205d5359b7ecb778c3d784ef1bb72d34/cf2/f70a7ebc6c464a08ab00a25e137bbc72.c09b5f7c44d4a83602090cbccc082b61->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf2/f70a7ebc6c464a08ab00a25e137bbc72-top] to archive 2023-04-12 02:55:43,636 DEBUG [StoreCloser-testRecoveryAndDoubleExecution,41,1681268138347.205d5359b7ecb778c3d784ef1bb72d34.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-12 02:55:43,637 DEBUG [StoreCloser-testRecoveryAndDoubleExecution,41,1681268138347.205d5359b7ecb778c3d784ef1bb72d34.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/205d5359b7ecb778c3d784ef1bb72d34/cf2/5b22fe6156a04c8c9486fd6f189bf4d7.c09b5f7c44d4a83602090cbccc082b61 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testRecoveryAndDoubleExecution/205d5359b7ecb778c3d784ef1bb72d34/cf2/5b22fe6156a04c8c9486fd6f189bf4d7.c09b5f7c44d4a83602090cbccc082b61 2023-04-12 02:55:43,638 DEBUG [StoreCloser-testRecoveryAndDoubleExecution,41,1681268138347.205d5359b7ecb778c3d784ef1bb72d34.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/205d5359b7ecb778c3d784ef1bb72d34/cf2/a4e6a82328b24916838ccb970ddd7704.c09b5f7c44d4a83602090cbccc082b61 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testRecoveryAndDoubleExecution/205d5359b7ecb778c3d784ef1bb72d34/cf2/a4e6a82328b24916838ccb970ddd7704.c09b5f7c44d4a83602090cbccc082b61 2023-04-12 02:55:43,640 DEBUG [StoreCloser-testRecoveryAndDoubleExecution,41,1681268138347.205d5359b7ecb778c3d784ef1bb72d34.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/205d5359b7ecb778c3d784ef1bb72d34/cf2/fb8ce8dc74bd47fabd120f6f688962d5.c09b5f7c44d4a83602090cbccc082b61 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testRecoveryAndDoubleExecution/205d5359b7ecb778c3d784ef1bb72d34/cf2/fb8ce8dc74bd47fabd120f6f688962d5.c09b5f7c44d4a83602090cbccc082b61 2023-04-12 02:55:43,641 DEBUG [StoreCloser-testRecoveryAndDoubleExecution,41,1681268138347.205d5359b7ecb778c3d784ef1bb72d34.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/205d5359b7ecb778c3d784ef1bb72d34/cf2/fe343f29642942069fb8e38396f28358.c09b5f7c44d4a83602090cbccc082b61 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testRecoveryAndDoubleExecution/205d5359b7ecb778c3d784ef1bb72d34/cf2/fe343f29642942069fb8e38396f28358.c09b5f7c44d4a83602090cbccc082b61 2023-04-12 02:55:43,642 DEBUG [StoreCloser-testRecoveryAndDoubleExecution,41,1681268138347.205d5359b7ecb778c3d784ef1bb72d34.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/205d5359b7ecb778c3d784ef1bb72d34/cf2/1b3fc837d85449cba8b914aa7f18f0fe.c09b5f7c44d4a83602090cbccc082b61 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testRecoveryAndDoubleExecution/205d5359b7ecb778c3d784ef1bb72d34/cf2/1b3fc837d85449cba8b914aa7f18f0fe.c09b5f7c44d4a83602090cbccc082b61 2023-04-12 02:55:43,643 DEBUG [StoreCloser-testRecoveryAndDoubleExecution,41,1681268138347.205d5359b7ecb778c3d784ef1bb72d34.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/205d5359b7ecb778c3d784ef1bb72d34/cf2/a7d4784e93134ef2943643862fd08463.c09b5f7c44d4a83602090cbccc082b61 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testRecoveryAndDoubleExecution/205d5359b7ecb778c3d784ef1bb72d34/cf2/a7d4784e93134ef2943643862fd08463.c09b5f7c44d4a83602090cbccc082b61 2023-04-12 02:55:43,644 DEBUG [StoreCloser-testRecoveryAndDoubleExecution,41,1681268138347.205d5359b7ecb778c3d784ef1bb72d34.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/205d5359b7ecb778c3d784ef1bb72d34/cf2/f70a7ebc6c464a08ab00a25e137bbc72.c09b5f7c44d4a83602090cbccc082b61 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testRecoveryAndDoubleExecution/205d5359b7ecb778c3d784ef1bb72d34/cf2/f70a7ebc6c464a08ab00a25e137bbc72.c09b5f7c44d4a83602090cbccc082b61 2023-04-12 02:55:43,654 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/205d5359b7ecb778c3d784ef1bb72d34/recovered.edits/90.seqid, newMaxSeqId=90, maxSeqId=85 2023-04-12 02:55:43,655 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro 2023-04-12 02:55:43,655 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed testRecoveryAndDoubleExecution,41,1681268138347.205d5359b7ecb778c3d784ef1bb72d34. 2023-04-12 02:55:43,655 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 205d5359b7ecb778c3d784ef1bb72d34: 2023-04-12 02:55:43,658 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 205d5359b7ecb778c3d784ef1bb72d34 2023-04-12 02:55:43,659 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 25b84f3651d3c0adac7c0275b4cb1e0c 2023-04-12 02:55:43,659 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 25b84f3651d3c0adac7c0275b4cb1e0c, disabling compactions & flushes 2023-04-12 02:55:43,659 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region testRecoveryAndDoubleExecution,,1681268138347.25b84f3651d3c0adac7c0275b4cb1e0c. 2023-04-12 02:55:43,659 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testRecoveryAndDoubleExecution,,1681268138347.25b84f3651d3c0adac7c0275b4cb1e0c. 2023-04-12 02:55:43,659 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on testRecoveryAndDoubleExecution,,1681268138347.25b84f3651d3c0adac7c0275b4cb1e0c. after waiting 0 ms 2023-04-12 02:55:43,659 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region testRecoveryAndDoubleExecution,,1681268138347.25b84f3651d3c0adac7c0275b4cb1e0c. 2023-04-12 02:55:43,659 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=121 updating hbase:meta row=205d5359b7ecb778c3d784ef1bb72d34, regionState=CLOSED 2023-04-12 02:55:43,659 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRecoveryAndDoubleExecution,41,1681268138347.205d5359b7ecb778c3d784ef1bb72d34.","families":{"info":[{"qualifier":"regioninfo","vlen":66,"tag":[],"timestamp":"1681268143659"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681268143659"}]},"ts":"1681268143659"} 2023-04-12 02:55:43,664 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=122, resume processing ppid=121 2023-04-12 02:55:43,664 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=122, ppid=121, state=SUCCESS; CloseRegionProcedure 205d5359b7ecb778c3d784ef1bb72d34, server=jenkins-hbase4.apache.org,43091,1681268031804 in 193 msec 2023-04-12 02:55:43,665 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=121, ppid=119, state=SUCCESS; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=205d5359b7ecb778c3d784ef1bb72d34, UNASSIGN in 198 msec 2023-04-12 02:55:43,668 DEBUG [StoreCloser-testRecoveryAndDoubleExecution,,1681268138347.25b84f3651d3c0adac7c0275b4cb1e0c.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/25b84f3651d3c0adac7c0275b4cb1e0c/cf1/a79c7cc5c59c4f6bbe6a6a584d824f36.c09b5f7c44d4a83602090cbccc082b61->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf1/a79c7cc5c59c4f6bbe6a6a584d824f36-bottom, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/25b84f3651d3c0adac7c0275b4cb1e0c/cf1/63025bd542924f2caced3f22c416a5a1.c09b5f7c44d4a83602090cbccc082b61->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf1/63025bd542924f2caced3f22c416a5a1-bottom, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/25b84f3651d3c0adac7c0275b4cb1e0c/cf1/9d5ec6ededfd4de8b6d44e635d18346c.c09b5f7c44d4a83602090cbccc082b61->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf1/9d5ec6ededfd4de8b6d44e635d18346c-bottom, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/25b84f3651d3c0adac7c0275b4cb1e0c/cf1/001e134c60444ac2b388f9df6bf5670b.c09b5f7c44d4a83602090cbccc082b61->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf1/001e134c60444ac2b388f9df6bf5670b-bottom, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/25b84f3651d3c0adac7c0275b4cb1e0c/cf1/c14bd2fe66004d6d842c72f4b2a4ee90.c09b5f7c44d4a83602090cbccc082b61->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf1/c14bd2fe66004d6d842c72f4b2a4ee90-bottom, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/25b84f3651d3c0adac7c0275b4cb1e0c/cf1/1373dcc8adeb4ae4b9929ef46fbb4980.c09b5f7c44d4a83602090cbccc082b61->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf1/1373dcc8adeb4ae4b9929ef46fbb4980-bottom, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/25b84f3651d3c0adac7c0275b4cb1e0c/cf1/e62dda4fe11148e88206bdd397ca64c0.c09b5f7c44d4a83602090cbccc082b61->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf1/e62dda4fe11148e88206bdd397ca64c0-bottom] to archive 2023-04-12 02:55:43,669 DEBUG [StoreCloser-testRecoveryAndDoubleExecution,,1681268138347.25b84f3651d3c0adac7c0275b4cb1e0c.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-12 02:55:43,672 DEBUG [StoreCloser-testRecoveryAndDoubleExecution,,1681268138347.25b84f3651d3c0adac7c0275b4cb1e0c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/25b84f3651d3c0adac7c0275b4cb1e0c/cf1/a79c7cc5c59c4f6bbe6a6a584d824f36.c09b5f7c44d4a83602090cbccc082b61 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testRecoveryAndDoubleExecution/25b84f3651d3c0adac7c0275b4cb1e0c/cf1/a79c7cc5c59c4f6bbe6a6a584d824f36.c09b5f7c44d4a83602090cbccc082b61 2023-04-12 02:55:43,673 DEBUG [StoreCloser-testRecoveryAndDoubleExecution,,1681268138347.25b84f3651d3c0adac7c0275b4cb1e0c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/25b84f3651d3c0adac7c0275b4cb1e0c/cf1/63025bd542924f2caced3f22c416a5a1.c09b5f7c44d4a83602090cbccc082b61 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testRecoveryAndDoubleExecution/25b84f3651d3c0adac7c0275b4cb1e0c/cf1/63025bd542924f2caced3f22c416a5a1.c09b5f7c44d4a83602090cbccc082b61 2023-04-12 02:55:43,674 DEBUG [StoreCloser-testRecoveryAndDoubleExecution,,1681268138347.25b84f3651d3c0adac7c0275b4cb1e0c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/25b84f3651d3c0adac7c0275b4cb1e0c/cf1/9d5ec6ededfd4de8b6d44e635d18346c.c09b5f7c44d4a83602090cbccc082b61 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testRecoveryAndDoubleExecution/25b84f3651d3c0adac7c0275b4cb1e0c/cf1/9d5ec6ededfd4de8b6d44e635d18346c.c09b5f7c44d4a83602090cbccc082b61 2023-04-12 02:55:43,676 DEBUG [StoreCloser-testRecoveryAndDoubleExecution,,1681268138347.25b84f3651d3c0adac7c0275b4cb1e0c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/25b84f3651d3c0adac7c0275b4cb1e0c/cf1/001e134c60444ac2b388f9df6bf5670b.c09b5f7c44d4a83602090cbccc082b61 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testRecoveryAndDoubleExecution/25b84f3651d3c0adac7c0275b4cb1e0c/cf1/001e134c60444ac2b388f9df6bf5670b.c09b5f7c44d4a83602090cbccc082b61 2023-04-12 02:55:43,678 DEBUG [StoreCloser-testRecoveryAndDoubleExecution,,1681268138347.25b84f3651d3c0adac7c0275b4cb1e0c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/25b84f3651d3c0adac7c0275b4cb1e0c/cf1/c14bd2fe66004d6d842c72f4b2a4ee90.c09b5f7c44d4a83602090cbccc082b61 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testRecoveryAndDoubleExecution/25b84f3651d3c0adac7c0275b4cb1e0c/cf1/c14bd2fe66004d6d842c72f4b2a4ee90.c09b5f7c44d4a83602090cbccc082b61 2023-04-12 02:55:43,679 DEBUG [StoreCloser-testRecoveryAndDoubleExecution,,1681268138347.25b84f3651d3c0adac7c0275b4cb1e0c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/25b84f3651d3c0adac7c0275b4cb1e0c/cf1/1373dcc8adeb4ae4b9929ef46fbb4980.c09b5f7c44d4a83602090cbccc082b61 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testRecoveryAndDoubleExecution/25b84f3651d3c0adac7c0275b4cb1e0c/cf1/1373dcc8adeb4ae4b9929ef46fbb4980.c09b5f7c44d4a83602090cbccc082b61 2023-04-12 02:55:43,680 DEBUG [StoreCloser-testRecoveryAndDoubleExecution,,1681268138347.25b84f3651d3c0adac7c0275b4cb1e0c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/25b84f3651d3c0adac7c0275b4cb1e0c/cf1/e62dda4fe11148e88206bdd397ca64c0.c09b5f7c44d4a83602090cbccc082b61 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testRecoveryAndDoubleExecution/25b84f3651d3c0adac7c0275b4cb1e0c/cf1/e62dda4fe11148e88206bdd397ca64c0.c09b5f7c44d4a83602090cbccc082b61 2023-04-12 02:55:43,685 DEBUG [StoreCloser-testRecoveryAndDoubleExecution,,1681268138347.25b84f3651d3c0adac7c0275b4cb1e0c.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/25b84f3651d3c0adac7c0275b4cb1e0c/cf2/5b22fe6156a04c8c9486fd6f189bf4d7.c09b5f7c44d4a83602090cbccc082b61->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf2/5b22fe6156a04c8c9486fd6f189bf4d7-bottom, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/25b84f3651d3c0adac7c0275b4cb1e0c/cf2/a4e6a82328b24916838ccb970ddd7704.c09b5f7c44d4a83602090cbccc082b61->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf2/a4e6a82328b24916838ccb970ddd7704-bottom, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/25b84f3651d3c0adac7c0275b4cb1e0c/cf2/fb8ce8dc74bd47fabd120f6f688962d5.c09b5f7c44d4a83602090cbccc082b61->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf2/fb8ce8dc74bd47fabd120f6f688962d5-bottom, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/25b84f3651d3c0adac7c0275b4cb1e0c/cf2/fe343f29642942069fb8e38396f28358.c09b5f7c44d4a83602090cbccc082b61->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf2/fe343f29642942069fb8e38396f28358-bottom, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/25b84f3651d3c0adac7c0275b4cb1e0c/cf2/1b3fc837d85449cba8b914aa7f18f0fe.c09b5f7c44d4a83602090cbccc082b61->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf2/1b3fc837d85449cba8b914aa7f18f0fe-bottom, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/25b84f3651d3c0adac7c0275b4cb1e0c/cf2/a7d4784e93134ef2943643862fd08463.c09b5f7c44d4a83602090cbccc082b61->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf2/a7d4784e93134ef2943643862fd08463-bottom, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/25b84f3651d3c0adac7c0275b4cb1e0c/cf2/f70a7ebc6c464a08ab00a25e137bbc72.c09b5f7c44d4a83602090cbccc082b61->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf2/f70a7ebc6c464a08ab00a25e137bbc72-bottom] to archive 2023-04-12 02:55:43,685 DEBUG [StoreCloser-testRecoveryAndDoubleExecution,,1681268138347.25b84f3651d3c0adac7c0275b4cb1e0c.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-12 02:55:43,689 DEBUG [StoreCloser-testRecoveryAndDoubleExecution,,1681268138347.25b84f3651d3c0adac7c0275b4cb1e0c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/25b84f3651d3c0adac7c0275b4cb1e0c/cf2/5b22fe6156a04c8c9486fd6f189bf4d7.c09b5f7c44d4a83602090cbccc082b61 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testRecoveryAndDoubleExecution/25b84f3651d3c0adac7c0275b4cb1e0c/cf2/5b22fe6156a04c8c9486fd6f189bf4d7.c09b5f7c44d4a83602090cbccc082b61 2023-04-12 02:55:43,690 DEBUG [StoreCloser-testRecoveryAndDoubleExecution,,1681268138347.25b84f3651d3c0adac7c0275b4cb1e0c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/25b84f3651d3c0adac7c0275b4cb1e0c/cf2/a4e6a82328b24916838ccb970ddd7704.c09b5f7c44d4a83602090cbccc082b61 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testRecoveryAndDoubleExecution/25b84f3651d3c0adac7c0275b4cb1e0c/cf2/a4e6a82328b24916838ccb970ddd7704.c09b5f7c44d4a83602090cbccc082b61 2023-04-12 02:55:43,691 DEBUG [StoreCloser-testRecoveryAndDoubleExecution,,1681268138347.25b84f3651d3c0adac7c0275b4cb1e0c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/25b84f3651d3c0adac7c0275b4cb1e0c/cf2/fb8ce8dc74bd47fabd120f6f688962d5.c09b5f7c44d4a83602090cbccc082b61 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testRecoveryAndDoubleExecution/25b84f3651d3c0adac7c0275b4cb1e0c/cf2/fb8ce8dc74bd47fabd120f6f688962d5.c09b5f7c44d4a83602090cbccc082b61 2023-04-12 02:55:43,692 DEBUG [StoreCloser-testRecoveryAndDoubleExecution,,1681268138347.25b84f3651d3c0adac7c0275b4cb1e0c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/25b84f3651d3c0adac7c0275b4cb1e0c/cf2/fe343f29642942069fb8e38396f28358.c09b5f7c44d4a83602090cbccc082b61 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testRecoveryAndDoubleExecution/25b84f3651d3c0adac7c0275b4cb1e0c/cf2/fe343f29642942069fb8e38396f28358.c09b5f7c44d4a83602090cbccc082b61 2023-04-12 02:55:43,693 DEBUG [StoreCloser-testRecoveryAndDoubleExecution,,1681268138347.25b84f3651d3c0adac7c0275b4cb1e0c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/25b84f3651d3c0adac7c0275b4cb1e0c/cf2/1b3fc837d85449cba8b914aa7f18f0fe.c09b5f7c44d4a83602090cbccc082b61 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testRecoveryAndDoubleExecution/25b84f3651d3c0adac7c0275b4cb1e0c/cf2/1b3fc837d85449cba8b914aa7f18f0fe.c09b5f7c44d4a83602090cbccc082b61 2023-04-12 02:55:43,694 DEBUG [StoreCloser-testRecoveryAndDoubleExecution,,1681268138347.25b84f3651d3c0adac7c0275b4cb1e0c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/25b84f3651d3c0adac7c0275b4cb1e0c/cf2/a7d4784e93134ef2943643862fd08463.c09b5f7c44d4a83602090cbccc082b61 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testRecoveryAndDoubleExecution/25b84f3651d3c0adac7c0275b4cb1e0c/cf2/a7d4784e93134ef2943643862fd08463.c09b5f7c44d4a83602090cbccc082b61 2023-04-12 02:55:43,696 DEBUG [StoreCloser-testRecoveryAndDoubleExecution,,1681268138347.25b84f3651d3c0adac7c0275b4cb1e0c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/25b84f3651d3c0adac7c0275b4cb1e0c/cf2/f70a7ebc6c464a08ab00a25e137bbc72.c09b5f7c44d4a83602090cbccc082b61 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testRecoveryAndDoubleExecution/25b84f3651d3c0adac7c0275b4cb1e0c/cf2/f70a7ebc6c464a08ab00a25e137bbc72.c09b5f7c44d4a83602090cbccc082b61 2023-04-12 02:55:43,700 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testRecoveryAndDoubleExecution/25b84f3651d3c0adac7c0275b4cb1e0c/recovered.edits/90.seqid, newMaxSeqId=90, maxSeqId=85 2023-04-12 02:55:43,701 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro 2023-04-12 02:55:43,701 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed testRecoveryAndDoubleExecution,,1681268138347.25b84f3651d3c0adac7c0275b4cb1e0c. 2023-04-12 02:55:43,701 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 25b84f3651d3c0adac7c0275b4cb1e0c: 2023-04-12 02:55:43,709 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 25b84f3651d3c0adac7c0275b4cb1e0c 2023-04-12 02:55:43,709 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=120 updating hbase:meta row=25b84f3651d3c0adac7c0275b4cb1e0c, regionState=CLOSED 2023-04-12 02:55:43,710 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRecoveryAndDoubleExecution,,1681268138347.25b84f3651d3c0adac7c0275b4cb1e0c.","families":{"info":[{"qualifier":"regioninfo","vlen":66,"tag":[],"timestamp":"1681268143709"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681268143709"}]},"ts":"1681268143709"} 2023-04-12 02:55:43,712 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=123, resume processing ppid=120 2023-04-12 02:55:43,713 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=123, ppid=120, state=SUCCESS; CloseRegionProcedure 25b84f3651d3c0adac7c0275b4cb1e0c, server=jenkins-hbase4.apache.org,43091,1681268031804 in 240 msec 2023-04-12 02:55:43,714 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=120, resume processing ppid=119 2023-04-12 02:55:43,714 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=120, ppid=119, state=SUCCESS; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=25b84f3651d3c0adac7c0275b4cb1e0c, UNASSIGN in 247 msec 2023-04-12 02:55:43,715 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRecoveryAndDoubleExecution","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681268143715"}]},"ts":"1681268143715"} 2023-04-12 02:55:43,716 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testRecoveryAndDoubleExecution, state=DISABLED in hbase:meta 2023-04-12 02:55:43,718 INFO [PEWorker-1] procedure.DisableTableProcedure(305): Set testRecoveryAndDoubleExecution to state=DISABLED 2023-04-12 02:55:43,720 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=119, state=SUCCESS; DisableTableProcedure table=testRecoveryAndDoubleExecution in 263 msec 2023-04-12 02:55:43,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] master.MasterRpcServices(1227): Checking to see if procedure is done pid=119 2023-04-12 02:55:43,765 INFO [Listener at localhost/46667] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testRecoveryAndDoubleExecution, procId: 119 completed 2023-04-12 02:55:43,766 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete testRecoveryAndDoubleExecution 2023-04-12 02:55:43,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] procedure2.ProcedureExecutor(1029): Stored pid=124, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=testRecoveryAndDoubleExecution 2023-04-12 02:55:43,768 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=124, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=testRecoveryAndDoubleExecution 2023-04-12 02:55:43,769 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] master.MasterRpcServices(1227): Checking to see if procedure is done pid=124 2023-04-12 02:55:43,769 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=124, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=testRecoveryAndDoubleExecution 2023-04-12 02:55:43,774 DEBUG [HFileArchiver-13] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61 2023-04-12 02:55:43,774 DEBUG [HFileArchiver-14] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testRecoveryAndDoubleExecution/25b84f3651d3c0adac7c0275b4cb1e0c 2023-04-12 02:55:43,774 DEBUG [HFileArchiver-15] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testRecoveryAndDoubleExecution/205d5359b7ecb778c3d784ef1bb72d34 2023-04-12 02:55:43,778 DEBUG [HFileArchiver-13] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf1, FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf2, FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/recovered.edits] 2023-04-12 02:55:43,779 DEBUG [HFileArchiver-14] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testRecoveryAndDoubleExecution/25b84f3651d3c0adac7c0275b4cb1e0c/25b84f3651d3c0adac7c0275b4cb1e0c, FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testRecoveryAndDoubleExecution/25b84f3651d3c0adac7c0275b4cb1e0c/cf1, FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testRecoveryAndDoubleExecution/25b84f3651d3c0adac7c0275b4cb1e0c/cf2, FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testRecoveryAndDoubleExecution/25b84f3651d3c0adac7c0275b4cb1e0c/recovered.edits] 2023-04-12 02:55:43,779 DEBUG [HFileArchiver-15] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testRecoveryAndDoubleExecution/205d5359b7ecb778c3d784ef1bb72d34/205d5359b7ecb778c3d784ef1bb72d34, FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testRecoveryAndDoubleExecution/205d5359b7ecb778c3d784ef1bb72d34/cf1, FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testRecoveryAndDoubleExecution/205d5359b7ecb778c3d784ef1bb72d34/cf2, FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testRecoveryAndDoubleExecution/205d5359b7ecb778c3d784ef1bb72d34/recovered.edits] 2023-04-12 02:55:43,783 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf1/001e134c60444ac2b388f9df6bf5670b to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf1/001e134c60444ac2b388f9df6bf5670b 2023-04-12 02:55:43,784 DEBUG [HFileArchiver-15] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testRecoveryAndDoubleExecution/205d5359b7ecb778c3d784ef1bb72d34/205d5359b7ecb778c3d784ef1bb72d34/.regioninfo to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testRecoveryAndDoubleExecution/205d5359b7ecb778c3d784ef1bb72d34/205d5359b7ecb778c3d784ef1bb72d34/.regioninfo 2023-04-12 02:55:43,785 DEBUG [HFileArchiver-14] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testRecoveryAndDoubleExecution/25b84f3651d3c0adac7c0275b4cb1e0c/25b84f3651d3c0adac7c0275b4cb1e0c/.regioninfo to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testRecoveryAndDoubleExecution/25b84f3651d3c0adac7c0275b4cb1e0c/25b84f3651d3c0adac7c0275b4cb1e0c/.regioninfo 2023-04-12 02:55:43,786 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf1/1373dcc8adeb4ae4b9929ef46fbb4980 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf1/1373dcc8adeb4ae4b9929ef46fbb4980 2023-04-12 02:55:43,788 DEBUG [HFileArchiver-15] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testRecoveryAndDoubleExecution/205d5359b7ecb778c3d784ef1bb72d34/205d5359b7ecb778c3d784ef1bb72d34/cf1/001e134c60444ac2b388f9df6bf5670b.c09b5f7c44d4a83602090cbccc082b61 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testRecoveryAndDoubleExecution/205d5359b7ecb778c3d784ef1bb72d34/205d5359b7ecb778c3d784ef1bb72d34/cf1/001e134c60444ac2b388f9df6bf5670b.c09b5f7c44d4a83602090cbccc082b61 2023-04-12 02:55:43,789 DEBUG [HFileArchiver-14] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testRecoveryAndDoubleExecution/25b84f3651d3c0adac7c0275b4cb1e0c/25b84f3651d3c0adac7c0275b4cb1e0c/cf1/001e134c60444ac2b388f9df6bf5670b.c09b5f7c44d4a83602090cbccc082b61 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testRecoveryAndDoubleExecution/25b84f3651d3c0adac7c0275b4cb1e0c/25b84f3651d3c0adac7c0275b4cb1e0c/cf1/001e134c60444ac2b388f9df6bf5670b.c09b5f7c44d4a83602090cbccc082b61 2023-04-12 02:55:43,789 DEBUG [HFileArchiver-15] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testRecoveryAndDoubleExecution/205d5359b7ecb778c3d784ef1bb72d34/205d5359b7ecb778c3d784ef1bb72d34/cf1/1373dcc8adeb4ae4b9929ef46fbb4980.c09b5f7c44d4a83602090cbccc082b61 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testRecoveryAndDoubleExecution/205d5359b7ecb778c3d784ef1bb72d34/205d5359b7ecb778c3d784ef1bb72d34/cf1/1373dcc8adeb4ae4b9929ef46fbb4980.c09b5f7c44d4a83602090cbccc082b61 2023-04-12 02:55:43,789 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf1/63025bd542924f2caced3f22c416a5a1 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf1/63025bd542924f2caced3f22c416a5a1 2023-04-12 02:55:43,791 DEBUG [HFileArchiver-14] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testRecoveryAndDoubleExecution/25b84f3651d3c0adac7c0275b4cb1e0c/25b84f3651d3c0adac7c0275b4cb1e0c/cf1/1373dcc8adeb4ae4b9929ef46fbb4980.c09b5f7c44d4a83602090cbccc082b61 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testRecoveryAndDoubleExecution/25b84f3651d3c0adac7c0275b4cb1e0c/25b84f3651d3c0adac7c0275b4cb1e0c/cf1/1373dcc8adeb4ae4b9929ef46fbb4980.c09b5f7c44d4a83602090cbccc082b61 2023-04-12 02:55:43,792 DEBUG [HFileArchiver-15] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testRecoveryAndDoubleExecution/205d5359b7ecb778c3d784ef1bb72d34/205d5359b7ecb778c3d784ef1bb72d34/cf1/63025bd542924f2caced3f22c416a5a1.c09b5f7c44d4a83602090cbccc082b61 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testRecoveryAndDoubleExecution/205d5359b7ecb778c3d784ef1bb72d34/205d5359b7ecb778c3d784ef1bb72d34/cf1/63025bd542924f2caced3f22c416a5a1.c09b5f7c44d4a83602090cbccc082b61 2023-04-12 02:55:43,792 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf1/9d5ec6ededfd4de8b6d44e635d18346c to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf1/9d5ec6ededfd4de8b6d44e635d18346c 2023-04-12 02:55:43,793 DEBUG [HFileArchiver-14] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testRecoveryAndDoubleExecution/25b84f3651d3c0adac7c0275b4cb1e0c/25b84f3651d3c0adac7c0275b4cb1e0c/cf1/63025bd542924f2caced3f22c416a5a1.c09b5f7c44d4a83602090cbccc082b61 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testRecoveryAndDoubleExecution/25b84f3651d3c0adac7c0275b4cb1e0c/25b84f3651d3c0adac7c0275b4cb1e0c/cf1/63025bd542924f2caced3f22c416a5a1.c09b5f7c44d4a83602090cbccc082b61 2023-04-12 02:55:43,794 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf1/a79c7cc5c59c4f6bbe6a6a584d824f36 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf1/a79c7cc5c59c4f6bbe6a6a584d824f36 2023-04-12 02:55:43,794 DEBUG [HFileArchiver-15] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testRecoveryAndDoubleExecution/205d5359b7ecb778c3d784ef1bb72d34/205d5359b7ecb778c3d784ef1bb72d34/cf1/9d5ec6ededfd4de8b6d44e635d18346c.c09b5f7c44d4a83602090cbccc082b61 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testRecoveryAndDoubleExecution/205d5359b7ecb778c3d784ef1bb72d34/205d5359b7ecb778c3d784ef1bb72d34/cf1/9d5ec6ededfd4de8b6d44e635d18346c.c09b5f7c44d4a83602090cbccc082b61 2023-04-12 02:55:43,796 DEBUG [HFileArchiver-14] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testRecoveryAndDoubleExecution/25b84f3651d3c0adac7c0275b4cb1e0c/25b84f3651d3c0adac7c0275b4cb1e0c/cf1/9d5ec6ededfd4de8b6d44e635d18346c.c09b5f7c44d4a83602090cbccc082b61 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testRecoveryAndDoubleExecution/25b84f3651d3c0adac7c0275b4cb1e0c/25b84f3651d3c0adac7c0275b4cb1e0c/cf1/9d5ec6ededfd4de8b6d44e635d18346c.c09b5f7c44d4a83602090cbccc082b61 2023-04-12 02:55:43,798 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf1/c14bd2fe66004d6d842c72f4b2a4ee90 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf1/c14bd2fe66004d6d842c72f4b2a4ee90 2023-04-12 02:55:43,798 DEBUG [HFileArchiver-15] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testRecoveryAndDoubleExecution/205d5359b7ecb778c3d784ef1bb72d34/205d5359b7ecb778c3d784ef1bb72d34/cf1/a79c7cc5c59c4f6bbe6a6a584d824f36.c09b5f7c44d4a83602090cbccc082b61 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testRecoveryAndDoubleExecution/205d5359b7ecb778c3d784ef1bb72d34/205d5359b7ecb778c3d784ef1bb72d34/cf1/a79c7cc5c59c4f6bbe6a6a584d824f36.c09b5f7c44d4a83602090cbccc082b61 2023-04-12 02:55:43,798 DEBUG [HFileArchiver-14] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testRecoveryAndDoubleExecution/25b84f3651d3c0adac7c0275b4cb1e0c/25b84f3651d3c0adac7c0275b4cb1e0c/cf1/a79c7cc5c59c4f6bbe6a6a584d824f36.c09b5f7c44d4a83602090cbccc082b61 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testRecoveryAndDoubleExecution/25b84f3651d3c0adac7c0275b4cb1e0c/25b84f3651d3c0adac7c0275b4cb1e0c/cf1/a79c7cc5c59c4f6bbe6a6a584d824f36.c09b5f7c44d4a83602090cbccc082b61 2023-04-12 02:55:43,800 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf1/e62dda4fe11148e88206bdd397ca64c0 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf1/e62dda4fe11148e88206bdd397ca64c0 2023-04-12 02:55:43,800 DEBUG [HFileArchiver-15] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testRecoveryAndDoubleExecution/205d5359b7ecb778c3d784ef1bb72d34/205d5359b7ecb778c3d784ef1bb72d34/cf1/c14bd2fe66004d6d842c72f4b2a4ee90.c09b5f7c44d4a83602090cbccc082b61 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testRecoveryAndDoubleExecution/205d5359b7ecb778c3d784ef1bb72d34/205d5359b7ecb778c3d784ef1bb72d34/cf1/c14bd2fe66004d6d842c72f4b2a4ee90.c09b5f7c44d4a83602090cbccc082b61 2023-04-12 02:55:43,800 DEBUG [HFileArchiver-14] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testRecoveryAndDoubleExecution/25b84f3651d3c0adac7c0275b4cb1e0c/25b84f3651d3c0adac7c0275b4cb1e0c/cf1/c14bd2fe66004d6d842c72f4b2a4ee90.c09b5f7c44d4a83602090cbccc082b61 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testRecoveryAndDoubleExecution/25b84f3651d3c0adac7c0275b4cb1e0c/25b84f3651d3c0adac7c0275b4cb1e0c/cf1/c14bd2fe66004d6d842c72f4b2a4ee90.c09b5f7c44d4a83602090cbccc082b61 2023-04-12 02:55:43,802 DEBUG [HFileArchiver-14] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testRecoveryAndDoubleExecution/25b84f3651d3c0adac7c0275b4cb1e0c/25b84f3651d3c0adac7c0275b4cb1e0c/cf1/e62dda4fe11148e88206bdd397ca64c0.c09b5f7c44d4a83602090cbccc082b61 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testRecoveryAndDoubleExecution/25b84f3651d3c0adac7c0275b4cb1e0c/25b84f3651d3c0adac7c0275b4cb1e0c/cf1/e62dda4fe11148e88206bdd397ca64c0.c09b5f7c44d4a83602090cbccc082b61 2023-04-12 02:55:43,804 DEBUG [HFileArchiver-15] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testRecoveryAndDoubleExecution/205d5359b7ecb778c3d784ef1bb72d34/205d5359b7ecb778c3d784ef1bb72d34/cf1/e62dda4fe11148e88206bdd397ca64c0.c09b5f7c44d4a83602090cbccc082b61 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testRecoveryAndDoubleExecution/205d5359b7ecb778c3d784ef1bb72d34/205d5359b7ecb778c3d784ef1bb72d34/cf1/e62dda4fe11148e88206bdd397ca64c0.c09b5f7c44d4a83602090cbccc082b61 2023-04-12 02:55:43,806 DEBUG [HFileArchiver-14] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testRecoveryAndDoubleExecution/25b84f3651d3c0adac7c0275b4cb1e0c/25b84f3651d3c0adac7c0275b4cb1e0c/cf2/1b3fc837d85449cba8b914aa7f18f0fe.c09b5f7c44d4a83602090cbccc082b61 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testRecoveryAndDoubleExecution/25b84f3651d3c0adac7c0275b4cb1e0c/25b84f3651d3c0adac7c0275b4cb1e0c/cf2/1b3fc837d85449cba8b914aa7f18f0fe.c09b5f7c44d4a83602090cbccc082b61 2023-04-12 02:55:43,807 DEBUG [HFileArchiver-14] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testRecoveryAndDoubleExecution/25b84f3651d3c0adac7c0275b4cb1e0c/25b84f3651d3c0adac7c0275b4cb1e0c/cf2/5b22fe6156a04c8c9486fd6f189bf4d7.c09b5f7c44d4a83602090cbccc082b61 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testRecoveryAndDoubleExecution/25b84f3651d3c0adac7c0275b4cb1e0c/25b84f3651d3c0adac7c0275b4cb1e0c/cf2/5b22fe6156a04c8c9486fd6f189bf4d7.c09b5f7c44d4a83602090cbccc082b61 2023-04-12 02:55:43,808 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf2/1b3fc837d85449cba8b914aa7f18f0fe to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf2/1b3fc837d85449cba8b914aa7f18f0fe 2023-04-12 02:55:43,808 DEBUG [HFileArchiver-15] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testRecoveryAndDoubleExecution/205d5359b7ecb778c3d784ef1bb72d34/205d5359b7ecb778c3d784ef1bb72d34/cf2/1b3fc837d85449cba8b914aa7f18f0fe.c09b5f7c44d4a83602090cbccc082b61 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testRecoveryAndDoubleExecution/205d5359b7ecb778c3d784ef1bb72d34/205d5359b7ecb778c3d784ef1bb72d34/cf2/1b3fc837d85449cba8b914aa7f18f0fe.c09b5f7c44d4a83602090cbccc082b61 2023-04-12 02:55:43,811 DEBUG [HFileArchiver-14] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testRecoveryAndDoubleExecution/25b84f3651d3c0adac7c0275b4cb1e0c/25b84f3651d3c0adac7c0275b4cb1e0c/cf2/a4e6a82328b24916838ccb970ddd7704.c09b5f7c44d4a83602090cbccc082b61 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testRecoveryAndDoubleExecution/25b84f3651d3c0adac7c0275b4cb1e0c/25b84f3651d3c0adac7c0275b4cb1e0c/cf2/a4e6a82328b24916838ccb970ddd7704.c09b5f7c44d4a83602090cbccc082b61 2023-04-12 02:55:43,812 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf2/5b22fe6156a04c8c9486fd6f189bf4d7 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf2/5b22fe6156a04c8c9486fd6f189bf4d7 2023-04-12 02:55:43,813 DEBUG [HFileArchiver-15] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testRecoveryAndDoubleExecution/205d5359b7ecb778c3d784ef1bb72d34/205d5359b7ecb778c3d784ef1bb72d34/cf2/5b22fe6156a04c8c9486fd6f189bf4d7.c09b5f7c44d4a83602090cbccc082b61 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testRecoveryAndDoubleExecution/205d5359b7ecb778c3d784ef1bb72d34/205d5359b7ecb778c3d784ef1bb72d34/cf2/5b22fe6156a04c8c9486fd6f189bf4d7.c09b5f7c44d4a83602090cbccc082b61 2023-04-12 02:55:43,813 DEBUG [HFileArchiver-14] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testRecoveryAndDoubleExecution/25b84f3651d3c0adac7c0275b4cb1e0c/25b84f3651d3c0adac7c0275b4cb1e0c/cf2/a7d4784e93134ef2943643862fd08463.c09b5f7c44d4a83602090cbccc082b61 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testRecoveryAndDoubleExecution/25b84f3651d3c0adac7c0275b4cb1e0c/25b84f3651d3c0adac7c0275b4cb1e0c/cf2/a7d4784e93134ef2943643862fd08463.c09b5f7c44d4a83602090cbccc082b61 2023-04-12 02:55:43,814 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf2/a4e6a82328b24916838ccb970ddd7704 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf2/a4e6a82328b24916838ccb970ddd7704 2023-04-12 02:55:43,815 DEBUG [HFileArchiver-15] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testRecoveryAndDoubleExecution/205d5359b7ecb778c3d784ef1bb72d34/205d5359b7ecb778c3d784ef1bb72d34/cf2/a4e6a82328b24916838ccb970ddd7704.c09b5f7c44d4a83602090cbccc082b61 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testRecoveryAndDoubleExecution/205d5359b7ecb778c3d784ef1bb72d34/205d5359b7ecb778c3d784ef1bb72d34/cf2/a4e6a82328b24916838ccb970ddd7704.c09b5f7c44d4a83602090cbccc082b61 2023-04-12 02:55:43,815 DEBUG [HFileArchiver-14] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testRecoveryAndDoubleExecution/25b84f3651d3c0adac7c0275b4cb1e0c/25b84f3651d3c0adac7c0275b4cb1e0c/cf2/f70a7ebc6c464a08ab00a25e137bbc72.c09b5f7c44d4a83602090cbccc082b61 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testRecoveryAndDoubleExecution/25b84f3651d3c0adac7c0275b4cb1e0c/25b84f3651d3c0adac7c0275b4cb1e0c/cf2/f70a7ebc6c464a08ab00a25e137bbc72.c09b5f7c44d4a83602090cbccc082b61 2023-04-12 02:55:43,816 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf2/a7d4784e93134ef2943643862fd08463 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf2/a7d4784e93134ef2943643862fd08463 2023-04-12 02:55:43,821 DEBUG [HFileArchiver-14] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testRecoveryAndDoubleExecution/25b84f3651d3c0adac7c0275b4cb1e0c/25b84f3651d3c0adac7c0275b4cb1e0c/cf2/fb8ce8dc74bd47fabd120f6f688962d5.c09b5f7c44d4a83602090cbccc082b61 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testRecoveryAndDoubleExecution/25b84f3651d3c0adac7c0275b4cb1e0c/25b84f3651d3c0adac7c0275b4cb1e0c/cf2/fb8ce8dc74bd47fabd120f6f688962d5.c09b5f7c44d4a83602090cbccc082b61 2023-04-12 02:55:43,821 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf2/f70a7ebc6c464a08ab00a25e137bbc72 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf2/f70a7ebc6c464a08ab00a25e137bbc72 2023-04-12 02:55:43,822 DEBUG [HFileArchiver-14] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testRecoveryAndDoubleExecution/25b84f3651d3c0adac7c0275b4cb1e0c/25b84f3651d3c0adac7c0275b4cb1e0c/cf2/fe343f29642942069fb8e38396f28358.c09b5f7c44d4a83602090cbccc082b61 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testRecoveryAndDoubleExecution/25b84f3651d3c0adac7c0275b4cb1e0c/25b84f3651d3c0adac7c0275b4cb1e0c/cf2/fe343f29642942069fb8e38396f28358.c09b5f7c44d4a83602090cbccc082b61 2023-04-12 02:55:43,823 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf2/fb8ce8dc74bd47fabd120f6f688962d5 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf2/fb8ce8dc74bd47fabd120f6f688962d5 2023-04-12 02:55:43,825 DEBUG [HFileArchiver-15] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testRecoveryAndDoubleExecution/205d5359b7ecb778c3d784ef1bb72d34/205d5359b7ecb778c3d784ef1bb72d34/cf2/a7d4784e93134ef2943643862fd08463.c09b5f7c44d4a83602090cbccc082b61 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testRecoveryAndDoubleExecution/205d5359b7ecb778c3d784ef1bb72d34/205d5359b7ecb778c3d784ef1bb72d34/cf2/a7d4784e93134ef2943643862fd08463.c09b5f7c44d4a83602090cbccc082b61 2023-04-12 02:55:43,826 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf2/fe343f29642942069fb8e38396f28358 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/cf2/fe343f29642942069fb8e38396f28358 2023-04-12 02:55:43,826 DEBUG [HFileArchiver-14] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testRecoveryAndDoubleExecution/25b84f3651d3c0adac7c0275b4cb1e0c/cf1/8bb8c95221324ace979de6f9e8ee711c to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testRecoveryAndDoubleExecution/25b84f3651d3c0adac7c0275b4cb1e0c/cf1/8bb8c95221324ace979de6f9e8ee711c 2023-04-12 02:55:43,827 DEBUG [HFileArchiver-15] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testRecoveryAndDoubleExecution/205d5359b7ecb778c3d784ef1bb72d34/205d5359b7ecb778c3d784ef1bb72d34/cf2/f70a7ebc6c464a08ab00a25e137bbc72.c09b5f7c44d4a83602090cbccc082b61 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testRecoveryAndDoubleExecution/205d5359b7ecb778c3d784ef1bb72d34/205d5359b7ecb778c3d784ef1bb72d34/cf2/f70a7ebc6c464a08ab00a25e137bbc72.c09b5f7c44d4a83602090cbccc082b61 2023-04-12 02:55:43,829 DEBUG [HFileArchiver-15] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testRecoveryAndDoubleExecution/205d5359b7ecb778c3d784ef1bb72d34/205d5359b7ecb778c3d784ef1bb72d34/cf2/fb8ce8dc74bd47fabd120f6f688962d5.c09b5f7c44d4a83602090cbccc082b61 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testRecoveryAndDoubleExecution/205d5359b7ecb778c3d784ef1bb72d34/205d5359b7ecb778c3d784ef1bb72d34/cf2/fb8ce8dc74bd47fabd120f6f688962d5.c09b5f7c44d4a83602090cbccc082b61 2023-04-12 02:55:43,829 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/recovered.edits/85.seqid to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61/recovered.edits/85.seqid 2023-04-12 02:55:43,830 DEBUG [HFileArchiver-14] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testRecoveryAndDoubleExecution/25b84f3651d3c0adac7c0275b4cb1e0c/cf2/8de8d9f320ee4e34aac7d6d6450e1472 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testRecoveryAndDoubleExecution/25b84f3651d3c0adac7c0275b4cb1e0c/cf2/8de8d9f320ee4e34aac7d6d6450e1472 2023-04-12 02:55:43,830 DEBUG [HFileArchiver-13] backup.HFileArchiver(596): Deleted hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testRecoveryAndDoubleExecution/c09b5f7c44d4a83602090cbccc082b61 2023-04-12 02:55:43,832 DEBUG [HFileArchiver-15] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testRecoveryAndDoubleExecution/205d5359b7ecb778c3d784ef1bb72d34/205d5359b7ecb778c3d784ef1bb72d34/cf2/fe343f29642942069fb8e38396f28358.c09b5f7c44d4a83602090cbccc082b61 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testRecoveryAndDoubleExecution/205d5359b7ecb778c3d784ef1bb72d34/205d5359b7ecb778c3d784ef1bb72d34/cf2/fe343f29642942069fb8e38396f28358.c09b5f7c44d4a83602090cbccc082b61 2023-04-12 02:55:43,835 DEBUG [HFileArchiver-14] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testRecoveryAndDoubleExecution/25b84f3651d3c0adac7c0275b4cb1e0c/recovered.edits/90.seqid to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testRecoveryAndDoubleExecution/25b84f3651d3c0adac7c0275b4cb1e0c/recovered.edits/90.seqid 2023-04-12 02:55:43,835 DEBUG [HFileArchiver-15] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testRecoveryAndDoubleExecution/205d5359b7ecb778c3d784ef1bb72d34/cf1/2f03e2dbb8c9448b8297606a5f3ae63b to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testRecoveryAndDoubleExecution/205d5359b7ecb778c3d784ef1bb72d34/cf1/2f03e2dbb8c9448b8297606a5f3ae63b 2023-04-12 02:55:43,836 DEBUG [HFileArchiver-14] backup.HFileArchiver(596): Deleted hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testRecoveryAndDoubleExecution/25b84f3651d3c0adac7c0275b4cb1e0c 2023-04-12 02:55:43,838 DEBUG [HFileArchiver-15] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testRecoveryAndDoubleExecution/205d5359b7ecb778c3d784ef1bb72d34/cf2/df99aec444e54dbab97c30b0d5f46460 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testRecoveryAndDoubleExecution/205d5359b7ecb778c3d784ef1bb72d34/cf2/df99aec444e54dbab97c30b0d5f46460 2023-04-12 02:55:43,840 DEBUG [HFileArchiver-15] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testRecoveryAndDoubleExecution/205d5359b7ecb778c3d784ef1bb72d34/recovered.edits/90.seqid to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testRecoveryAndDoubleExecution/205d5359b7ecb778c3d784ef1bb72d34/recovered.edits/90.seqid 2023-04-12 02:55:43,841 DEBUG [HFileArchiver-15] backup.HFileArchiver(596): Deleted hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testRecoveryAndDoubleExecution/205d5359b7ecb778c3d784ef1bb72d34 2023-04-12 02:55:43,842 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived testRecoveryAndDoubleExecution regions 2023-04-12 02:55:43,845 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=124, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=testRecoveryAndDoubleExecution 2023-04-12 02:55:43,846 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 3 rows of testRecoveryAndDoubleExecution from hbase:meta 2023-04-12 02:55:43,849 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'testRecoveryAndDoubleExecution' descriptor. 2023-04-12 02:55:43,850 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=124, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=testRecoveryAndDoubleExecution 2023-04-12 02:55:43,850 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'testRecoveryAndDoubleExecution' from region states. 2023-04-12 02:55:43,850 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRecoveryAndDoubleExecution,,1681268136320.c09b5f7c44d4a83602090cbccc082b61.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1681268143850"}]},"ts":"9223372036854775807"} 2023-04-12 02:55:43,850 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRecoveryAndDoubleExecution,,1681268138347.25b84f3651d3c0adac7c0275b4cb1e0c.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1681268143850"}]},"ts":"9223372036854775807"} 2023-04-12 02:55:43,850 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRecoveryAndDoubleExecution,41,1681268138347.205d5359b7ecb778c3d784ef1bb72d34.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1681268143850"}]},"ts":"9223372036854775807"} 2023-04-12 02:55:43,852 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 3 regions from META 2023-04-12 02:55:43,852 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => c09b5f7c44d4a83602090cbccc082b61, NAME => 'testRecoveryAndDoubleExecution,,1681268136320.c09b5f7c44d4a83602090cbccc082b61.', STARTKEY => '', ENDKEY => '', OFFLINE => true, SPLIT => true}, {ENCODED => 25b84f3651d3c0adac7c0275b4cb1e0c, NAME => 'testRecoveryAndDoubleExecution,,1681268138347.25b84f3651d3c0adac7c0275b4cb1e0c.', STARTKEY => '', ENDKEY => '41'}, {ENCODED => 205d5359b7ecb778c3d784ef1bb72d34, NAME => 'testRecoveryAndDoubleExecution,41,1681268138347.205d5359b7ecb778c3d784ef1bb72d34.', STARTKEY => '41', ENDKEY => ''}] 2023-04-12 02:55:43,852 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'testRecoveryAndDoubleExecution' as deleted. 2023-04-12 02:55:43,852 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRecoveryAndDoubleExecution","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1681268143852"}]},"ts":"9223372036854775807"} 2023-04-12 02:55:43,854 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table testRecoveryAndDoubleExecution state from META 2023-04-12 02:55:43,855 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=124, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=testRecoveryAndDoubleExecution 2023-04-12 02:55:43,857 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=124, state=SUCCESS; DeleteTableProcedure table=testRecoveryAndDoubleExecution in 89 msec 2023-04-12 02:55:43,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] master.MasterRpcServices(1227): Checking to see if procedure is done pid=124 2023-04-12 02:55:43,870 INFO [Listener at localhost/46667] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:testRecoveryAndDoubleExecution, procId: 124 completed 2023-04-12 02:55:43,888 INFO [Listener at localhost/46667] hbase.ResourceChecker(175): after: master.assignment.TestSplitTableRegionProcedure#testRecoveryAndDoubleExecution Thread=375 (was 365) - Thread LEAK? -, OpenFileDescriptor=821 (was 736) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=350 (was 314) - SystemLoadAverage LEAK? -, ProcessCount=171 (was 172), AvailableMemoryMB=817 (was 836) 2023-04-12 02:55:43,906 INFO [Listener at localhost/46667] hbase.ResourceChecker(147): before: master.assignment.TestSplitTableRegionProcedure#testSplitTableRegionEmptyDaughter Thread=375, OpenFileDescriptor=821, MaxFileDescriptor=60000, SystemLoadAverage=350, ProcessCount=171, AvailableMemoryMB=817 2023-04-12 02:55:43,906 WARN [Listener at localhost/46667] procedure2.ProcedureTestingUtility(193): Set Kill before store update to: false 2023-04-12 02:55:43,909 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/balancer 2023-04-12 02:55:43,909 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] master.MasterRpcServices(492): Client=jenkins//172.31.14.131 set balanceSwitch=false 2023-04-12 02:55:43,911 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(1029): Stored pid=125, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=testSplitTableRegionEmptyDaughter 2023-04-12 02:55:43,912 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=125, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=testSplitTableRegionEmptyDaughter execute state=CREATE_TABLE_PRE_OPERATION 2023-04-12 02:55:43,913 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=125, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=testSplitTableRegionEmptyDaughter execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-12 02:55:43,914 DEBUG [HFileArchiver-16] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitTableRegionEmptyDaughter/7e32820b75c2980cc8d414d4b96996d4 2023-04-12 02:55:43,915 DEBUG [HFileArchiver-16] backup.HFileArchiver(153): Directory hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitTableRegionEmptyDaughter/7e32820b75c2980cc8d414d4b96996d4 empty. 2023-04-12 02:55:43,915 DEBUG [HFileArchiver-16] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitTableRegionEmptyDaughter/7e32820b75c2980cc8d414d4b96996d4 2023-04-12 02:55:43,915 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived testSplitTableRegionEmptyDaughter regions 2023-04-12 02:55:44,327 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitTableRegionEmptyDaughter/.tabledesc/.tableinfo.0000000001 2023-04-12 02:55:44,329 INFO [RegionOpenAndInit-testSplitTableRegionEmptyDaughter-pool-0] regionserver.HRegion(7675): creating {ENCODED => 7e32820b75c2980cc8d414d4b96996d4, NAME => 'testSplitTableRegionEmptyDaughter,,1681268143910.7e32820b75c2980cc8d414d4b96996d4.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='testSplitTableRegionEmptyDaughter', {NAME => 'cf1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'cf2', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp 2023-04-12 02:55:44,338 DEBUG [RegionOpenAndInit-testSplitTableRegionEmptyDaughter-pool-0] regionserver.HRegion(866): Instantiated testSplitTableRegionEmptyDaughter,,1681268143910.7e32820b75c2980cc8d414d4b96996d4.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-12 02:55:44,339 DEBUG [RegionOpenAndInit-testSplitTableRegionEmptyDaughter-pool-0] regionserver.HRegion(1604): Closing 7e32820b75c2980cc8d414d4b96996d4, disabling compactions & flushes 2023-04-12 02:55:44,339 INFO [RegionOpenAndInit-testSplitTableRegionEmptyDaughter-pool-0] regionserver.HRegion(1626): Closing region testSplitTableRegionEmptyDaughter,,1681268143910.7e32820b75c2980cc8d414d4b96996d4. 2023-04-12 02:55:44,339 DEBUG [RegionOpenAndInit-testSplitTableRegionEmptyDaughter-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testSplitTableRegionEmptyDaughter,,1681268143910.7e32820b75c2980cc8d414d4b96996d4. 2023-04-12 02:55:44,339 DEBUG [RegionOpenAndInit-testSplitTableRegionEmptyDaughter-pool-0] regionserver.HRegion(1714): Acquired close lock on testSplitTableRegionEmptyDaughter,,1681268143910.7e32820b75c2980cc8d414d4b96996d4. after waiting 0 ms 2023-04-12 02:55:44,339 DEBUG [RegionOpenAndInit-testSplitTableRegionEmptyDaughter-pool-0] regionserver.HRegion(1724): Updates disabled for region testSplitTableRegionEmptyDaughter,,1681268143910.7e32820b75c2980cc8d414d4b96996d4. 2023-04-12 02:55:44,339 INFO [RegionOpenAndInit-testSplitTableRegionEmptyDaughter-pool-0] regionserver.HRegion(1838): Closed testSplitTableRegionEmptyDaughter,,1681268143910.7e32820b75c2980cc8d414d4b96996d4. 2023-04-12 02:55:44,339 DEBUG [RegionOpenAndInit-testSplitTableRegionEmptyDaughter-pool-0] regionserver.HRegion(1558): Region close journal for 7e32820b75c2980cc8d414d4b96996d4: 2023-04-12 02:55:44,342 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=125, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=testSplitTableRegionEmptyDaughter execute state=CREATE_TABLE_ADD_TO_META 2023-04-12 02:55:44,342 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testSplitTableRegionEmptyDaughter,,1681268143910.7e32820b75c2980cc8d414d4b96996d4.","families":{"info":[{"qualifier":"regioninfo","vlen":67,"tag":[],"timestamp":"1681268144342"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681268144342"}]},"ts":"1681268144342"} 2023-04-12 02:55:44,343 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-12 02:55:44,344 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=125, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=testSplitTableRegionEmptyDaughter execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-12 02:55:44,344 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testSplitTableRegionEmptyDaughter","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681268144344"}]},"ts":"1681268144344"} 2023-04-12 02:55:44,345 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testSplitTableRegionEmptyDaughter, state=ENABLING in hbase:meta 2023-04-12 02:55:44,348 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-04-12 02:55:44,349 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-04-12 02:55:44,349 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-04-12 02:55:44,349 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-04-12 02:55:44,349 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-04-12 02:55:44,350 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=126, ppid=125, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSplitTableRegionEmptyDaughter, region=7e32820b75c2980cc8d414d4b96996d4, ASSIGN}] 2023-04-12 02:55:44,351 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=126, ppid=125, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSplitTableRegionEmptyDaughter, region=7e32820b75c2980cc8d414d4b96996d4, ASSIGN 2023-04-12 02:55:44,352 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=126, ppid=125, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testSplitTableRegionEmptyDaughter, region=7e32820b75c2980cc8d414d4b96996d4, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,43091,1681268031804; forceNewPlan=false, retain=false 2023-04-12 02:55:44,502 INFO [jenkins-hbase4:40281] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-04-12 02:55:44,503 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=126 updating hbase:meta row=7e32820b75c2980cc8d414d4b96996d4, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:44,504 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitTableRegionEmptyDaughter,,1681268143910.7e32820b75c2980cc8d414d4b96996d4.","families":{"info":[{"qualifier":"regioninfo","vlen":67,"tag":[],"timestamp":"1681268144503"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681268144503"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681268144503"}]},"ts":"1681268144503"} 2023-04-12 02:55:44,505 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=127, ppid=126, state=RUNNABLE; OpenRegionProcedure 7e32820b75c2980cc8d414d4b96996d4, server=jenkins-hbase4.apache.org,43091,1681268031804}] 2023-04-12 02:55:44,660 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testSplitTableRegionEmptyDaughter,,1681268143910.7e32820b75c2980cc8d414d4b96996d4. 2023-04-12 02:55:44,661 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 7e32820b75c2980cc8d414d4b96996d4, NAME => 'testSplitTableRegionEmptyDaughter,,1681268143910.7e32820b75c2980cc8d414d4b96996d4.', STARTKEY => '', ENDKEY => ''} 2023-04-12 02:55:44,661 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(174): System coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro loaded, priority=536870911. 2023-04-12 02:55:44,661 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testSplitTableRegionEmptyDaughter 7e32820b75c2980cc8d414d4b96996d4 2023-04-12 02:55:44,661 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated testSplitTableRegionEmptyDaughter,,1681268143910.7e32820b75c2980cc8d414d4b96996d4.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-12 02:55:44,661 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 7e32820b75c2980cc8d414d4b96996d4 2023-04-12 02:55:44,661 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 7e32820b75c2980cc8d414d4b96996d4 2023-04-12 02:55:44,663 INFO [StoreOpener-7e32820b75c2980cc8d414d4b96996d4-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf1 of region 7e32820b75c2980cc8d414d4b96996d4 2023-04-12 02:55:44,664 DEBUG [StoreOpener-7e32820b75c2980cc8d414d4b96996d4-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/7e32820b75c2980cc8d414d4b96996d4/cf1 2023-04-12 02:55:44,664 DEBUG [StoreOpener-7e32820b75c2980cc8d414d4b96996d4-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/7e32820b75c2980cc8d414d4b96996d4/cf1 2023-04-12 02:55:44,664 INFO [StoreOpener-7e32820b75c2980cc8d414d4b96996d4-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 7e32820b75c2980cc8d414d4b96996d4 columnFamilyName cf1 2023-04-12 02:55:44,665 INFO [StoreOpener-7e32820b75c2980cc8d414d4b96996d4-1] regionserver.HStore(310): Store=7e32820b75c2980cc8d414d4b96996d4/cf1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-12 02:55:44,665 INFO [StoreOpener-7e32820b75c2980cc8d414d4b96996d4-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf2 of region 7e32820b75c2980cc8d414d4b96996d4 2023-04-12 02:55:44,666 DEBUG [StoreOpener-7e32820b75c2980cc8d414d4b96996d4-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/7e32820b75c2980cc8d414d4b96996d4/cf2 2023-04-12 02:55:44,666 DEBUG [StoreOpener-7e32820b75c2980cc8d414d4b96996d4-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/7e32820b75c2980cc8d414d4b96996d4/cf2 2023-04-12 02:55:44,666 INFO [StoreOpener-7e32820b75c2980cc8d414d4b96996d4-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 7e32820b75c2980cc8d414d4b96996d4 columnFamilyName cf2 2023-04-12 02:55:44,667 INFO [StoreOpener-7e32820b75c2980cc8d414d4b96996d4-1] regionserver.HStore(310): Store=7e32820b75c2980cc8d414d4b96996d4/cf2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-12 02:55:44,668 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/7e32820b75c2980cc8d414d4b96996d4 2023-04-12 02:55:44,668 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/7e32820b75c2980cc8d414d4b96996d4 2023-04-12 02:55:44,670 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testSplitTableRegionEmptyDaughter descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-04-12 02:55:44,671 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 7e32820b75c2980cc8d414d4b96996d4 2023-04-12 02:55:44,672 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/7e32820b75c2980cc8d414d4b96996d4/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-12 02:55:44,673 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 7e32820b75c2980cc8d414d4b96996d4; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=72081545, jitterRate=0.07409872114658356}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-04-12 02:55:44,673 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 7e32820b75c2980cc8d414d4b96996d4: 2023-04-12 02:55:44,673 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testSplitTableRegionEmptyDaughter,,1681268143910.7e32820b75c2980cc8d414d4b96996d4., pid=127, masterSystemTime=1681268144658 2023-04-12 02:55:44,675 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testSplitTableRegionEmptyDaughter,,1681268143910.7e32820b75c2980cc8d414d4b96996d4. 2023-04-12 02:55:44,675 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testSplitTableRegionEmptyDaughter,,1681268143910.7e32820b75c2980cc8d414d4b96996d4. 2023-04-12 02:55:44,675 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=126 updating hbase:meta row=7e32820b75c2980cc8d414d4b96996d4, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:44,675 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testSplitTableRegionEmptyDaughter,,1681268143910.7e32820b75c2980cc8d414d4b96996d4.","families":{"info":[{"qualifier":"regioninfo","vlen":67,"tag":[],"timestamp":"1681268144675"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1681268144675"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681268144675"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681268144675"}]},"ts":"1681268144675"} 2023-04-12 02:55:44,679 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=127, resume processing ppid=126 2023-04-12 02:55:44,679 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=127, ppid=126, state=SUCCESS; OpenRegionProcedure 7e32820b75c2980cc8d414d4b96996d4, server=jenkins-hbase4.apache.org,43091,1681268031804 in 172 msec 2023-04-12 02:55:44,680 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=126, resume processing ppid=125 2023-04-12 02:55:44,681 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=126, ppid=125, state=SUCCESS; TransitRegionStateProcedure table=testSplitTableRegionEmptyDaughter, region=7e32820b75c2980cc8d414d4b96996d4, ASSIGN in 329 msec 2023-04-12 02:55:44,681 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=125, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=testSplitTableRegionEmptyDaughter execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-12 02:55:44,681 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testSplitTableRegionEmptyDaughter","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681268144681"}]},"ts":"1681268144681"} 2023-04-12 02:55:44,682 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testSplitTableRegionEmptyDaughter, state=ENABLED in hbase:meta 2023-04-12 02:55:44,685 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=125, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=testSplitTableRegionEmptyDaughter execute state=CREATE_TABLE_POST_OPERATION 2023-04-12 02:55:44,686 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=125, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionEmptyDaughter in 775 msec 2023-04-12 02:55:44,872 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-12 02:55:44,918 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 02:55:44,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] procedure.ProcedureCoordinator(165): Submitting procedure testSplitTableRegionEmptyDaughter 2023-04-12 02:55:44,919 INFO [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testSplitTableRegionEmptyDaughter' 2023-04-12 02:55:44,919 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:55:44,919 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testSplitTableRegionEmptyDaughter' starting 'acquire' 2023-04-12 02:55:44,919 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testSplitTableRegionEmptyDaughter', kicking off acquire phase on members. 2023-04-12 02:55:44,920 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:44,920 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:44,922 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:55:44,922 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:55:44,922 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:55:44,922 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:55:44,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:55:44,922 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:55:44,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:55:44,922 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:44,922 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:55:44,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:55:44,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:44,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:44,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:44,924 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:44,924 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 02:55:44,924 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:44,924 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:44,924 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:44,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 02:55:44,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:44,924 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionEmptyDaughter 2023-04-12 02:55:44,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 02:55:44,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:44,924 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionEmptyDaughter 2023-04-12 02:55:44,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 02:55:44,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:44,925 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionEmptyDaughter 2023-04-12 02:55:44,928 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionEmptyDaughter 2023-04-12 02:55:44,928 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionEmptyDaughter 2023-04-12 02:55:44,928 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionEmptyDaughter' with timeout 60000ms 2023-04-12 02:55:44,928 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:55:44,928 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionEmptyDaughter 2023-04-12 02:55:44,928 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionEmptyDaughter' starting 'acquire' stage 2023-04-12 02:55:44,928 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionEmptyDaughter' with timeout 60000ms 2023-04-12 02:55:44,928 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:55:44,928 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionEmptyDaughter' with timeout 60000ms 2023-04-12 02:55:44,928 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionEmptyDaughter' starting 'acquire' stage 2023-04-12 02:55:44,928 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionEmptyDaughter' locally acquired 2023-04-12 02:55:44,928 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34579,1681268031719' joining acquired barrier for procedure (testSplitTableRegionEmptyDaughter) in zk 2023-04-12 02:55:44,928 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:55:44,928 DEBUG [rs(jenkins-hbase4.apache.org,43091,1681268031804)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testSplitTableRegionEmptyDaughter,,1681268143910.7e32820b75c2980cc8d414d4b96996d4. 2023-04-12 02:55:44,928 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionEmptyDaughter' starting 'acquire' stage 2023-04-12 02:55:44,928 DEBUG [rs(jenkins-hbase4.apache.org,43091,1681268031804)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testSplitTableRegionEmptyDaughter,,1681268143910.7e32820b75c2980cc8d414d4b96996d4. started... 2023-04-12 02:55:44,928 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 02:55:44,929 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 02:55:44,930 INFO [rs(jenkins-hbase4.apache.org,43091,1681268031804)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 7e32820b75c2980cc8d414d4b96996d4 2/2 column families, dataSize=120 B heapSize=928 B 2023-04-12 02:55:44,928 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionEmptyDaughter' locally acquired 2023-04-12 02:55:44,930 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44679,1681268031772' joining acquired barrier for procedure (testSplitTableRegionEmptyDaughter) in zk 2023-04-12 02:55:44,932 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:44,933 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:44,933 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionEmptyDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 02:55:44,933 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:44,933 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:44,933 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionEmptyDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 02:55:44,947 INFO [rs(jenkins-hbase4.apache.org,43091,1681268031804)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=60 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/7e32820b75c2980cc8d414d4b96996d4/.tmp/cf1/abe57c09e0794d459254cbdc06a94880 2023-04-12 02:55:44,983 INFO [rs(jenkins-hbase4.apache.org,43091,1681268031804)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=60 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/7e32820b75c2980cc8d414d4b96996d4/.tmp/cf2/8cf94287d6174b3daefe82fefab38588 2023-04-12 02:55:44,988 DEBUG [rs(jenkins-hbase4.apache.org,43091,1681268031804)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/7e32820b75c2980cc8d414d4b96996d4/.tmp/cf1/abe57c09e0794d459254cbdc06a94880 as hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/7e32820b75c2980cc8d414d4b96996d4/cf1/abe57c09e0794d459254cbdc06a94880 2023-04-12 02:55:44,994 INFO [rs(jenkins-hbase4.apache.org,43091,1681268031804)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/7e32820b75c2980cc8d414d4b96996d4/cf1/abe57c09e0794d459254cbdc06a94880, entries=2, sequenceid=6, filesize=4.8 K 2023-04-12 02:55:44,994 DEBUG [rs(jenkins-hbase4.apache.org,43091,1681268031804)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/7e32820b75c2980cc8d414d4b96996d4/.tmp/cf2/8cf94287d6174b3daefe82fefab38588 as hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/7e32820b75c2980cc8d414d4b96996d4/cf2/8cf94287d6174b3daefe82fefab38588 2023-04-12 02:55:44,998 INFO [rs(jenkins-hbase4.apache.org,43091,1681268031804)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/7e32820b75c2980cc8d414d4b96996d4/cf2/8cf94287d6174b3daefe82fefab38588, entries=2, sequenceid=6, filesize=4.8 K 2023-04-12 02:55:44,999 INFO [rs(jenkins-hbase4.apache.org,43091,1681268031804)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~120 B/120, heapSize ~896 B/896, currentSize=0 B/0 for 7e32820b75c2980cc8d414d4b96996d4 in 70ms, sequenceid=6, compaction requested=false 2023-04-12 02:55:44,999 DEBUG [rs(jenkins-hbase4.apache.org,43091,1681268031804)-flush-proc-pool-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'testSplitTableRegionEmptyDaughter' 2023-04-12 02:55:44,999 DEBUG [rs(jenkins-hbase4.apache.org,43091,1681268031804)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7e32820b75c2980cc8d414d4b96996d4: 2023-04-12 02:55:45,000 DEBUG [rs(jenkins-hbase4.apache.org,43091,1681268031804)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testSplitTableRegionEmptyDaughter,,1681268143910.7e32820b75c2980cc8d414d4b96996d4. 2023-04-12 02:55:45,000 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 02:55:45,000 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 02:55:45,000 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:45,000 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionEmptyDaughter' locally acquired 2023-04-12 02:55:45,000 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43091,1681268031804' joining acquired barrier for procedure (testSplitTableRegionEmptyDaughter) in zk 2023-04-12 02:55:45,002 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:45,002 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:45,002 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionEmptyDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 02:55:45,003 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:45,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:45,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 02:55:45,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 02:55:45,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 02:55:45,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 02:55:45,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionEmptyDaughter 2023-04-12 02:55:45,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:45,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:45,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:45,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 02:55:45,007 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43091,1681268031804' joining acquired barrier for procedure 'testSplitTableRegionEmptyDaughter' on coordinator 2023-04-12 02:55:45,007 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4a6f410[Count = 0] remaining members to acquire global barrier 2023-04-12 02:55:45,007 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testSplitTableRegionEmptyDaughter' starting 'in-barrier' execution. 2023-04-12 02:55:45,007 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:45,008 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:45,009 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:45,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:45,009 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:45,009 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionEmptyDaughter' received 'reached' from coordinator. 2023-04-12 02:55:45,009 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionEmptyDaughter' locally completed 2023-04-12 02:55:45,009 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:45,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:45,009 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionEmptyDaughter' completed for member 'jenkins-hbase4.apache.org,34579,1681268031719' in zk 2023-04-12 02:55:45,009 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:45,009 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:45,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:45,009 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionEmptyDaughter' received 'reached' from coordinator. 2023-04-12 02:55:45,009 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionEmptyDaughter' locally completed 2023-04-12 02:55:45,009 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionEmptyDaughter' completed for member 'jenkins-hbase4.apache.org,44679,1681268031772' in zk 2023-04-12 02:55:45,009 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionEmptyDaughter' received 'reached' from coordinator. 2023-04-12 02:55:45,010 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionEmptyDaughter' locally completed 2023-04-12 02:55:45,010 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionEmptyDaughter' completed for member 'jenkins-hbase4.apache.org,43091,1681268031804' in zk 2023-04-12 02:55:45,011 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionEmptyDaughter' has notified controller of completion 2023-04-12 02:55:45,011 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:55:45,011 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionEmptyDaughter' completed. 2023-04-12 02:55:45,011 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:45,011 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 02:55:45,012 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:45,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:45,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 02:55:45,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 02:55:45,012 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionEmptyDaughter' has notified controller of completion 2023-04-12 02:55:45,012 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:55:45,012 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionEmptyDaughter' completed. 2023-04-12 02:55:45,013 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionEmptyDaughter' has notified controller of completion 2023-04-12 02:55:45,013 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:55:45,013 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionEmptyDaughter' completed. 2023-04-12 02:55:45,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 02:55:45,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 02:55:45,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionEmptyDaughter 2023-04-12 02:55:45,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:45,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:45,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:45,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 02:55:45,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionEmptyDaughter 2023-04-12 02:55:45,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:45,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:45,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:45,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testSplitTableRegionEmptyDaughter' member 'jenkins-hbase4.apache.org,43091,1681268031804': 2023-04-12 02:55:45,018 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43091,1681268031804' released barrier for procedure'testSplitTableRegionEmptyDaughter', counting down latch. Waiting for 0 more 2023-04-12 02:55:45,018 INFO [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testSplitTableRegionEmptyDaughter' execution completed 2023-04-12 02:55:45,018 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 02:55:45,018 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 02:55:45,018 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testSplitTableRegionEmptyDaughter 2023-04-12 02:55:45,018 INFO [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testSplitTableRegionEmptyDaughterincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 02:55:45,020 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:45,020 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:45,020 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:45,021 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:45,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:45,020 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:55:45,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:45,020 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:55:45,021 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:45,021 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:55:45,021 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:45,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:45,021 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:45,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:45,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 02:55:45,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 02:55:45,022 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:55:45,022 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:55:45,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:55:45,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:55:45,022 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:55:45,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 02:55:45,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:55:45,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:45,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:45,022 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:45,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:45,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionEmptyDaughter 2023-04-12 02:55:45,023 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:45,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 02:55:45,023 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:45,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionEmptyDaughter 2023-04-12 02:55:45,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:45,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:45,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:45,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 02:55:45,025 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:45,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionEmptyDaughter 2023-04-12 02:55:45,025 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:45,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:45,026 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:45,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:45,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:45,031 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:45,031 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:45,032 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:55:45,032 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:55:45,032 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:55:45,032 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:55:45,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:55:45,032 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:55:45,032 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:45,032 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:55:45,032 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:45,032 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:45,032 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:45,032 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:45,032 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:45,032 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:45,032 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:45,032 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:45,032 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testSplitTableRegionEmptyDaughter' 2023-04-12 02:55:45,032 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 02:55:45,032 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:55:45,032 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:55:45,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:55:45,032 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:55:45,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:55:45,033 DEBUG [Listener at localhost/46667] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testSplitTableRegionEmptyDaughter'' to complete. (max 20000 ms per retry) 2023-04-12 02:55:45,033 DEBUG [Listener at localhost/46667] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 02:55:45,032 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:55:45,032 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:55:45,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:55:45,033 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:55:45,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:55:45,033 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:55:45,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:55:45,133 DEBUG [Listener at localhost/46667] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 02:55:45,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 02:55:45,147 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 02:55:45,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] procedure.ProcedureCoordinator(143): Procedure testSplitTableRegionEmptyDaughter was in running list but was completed. Accepting new attempt. 2023-04-12 02:55:45,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] procedure.ProcedureCoordinator(165): Submitting procedure testSplitTableRegionEmptyDaughter 2023-04-12 02:55:45,149 INFO [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testSplitTableRegionEmptyDaughter' 2023-04-12 02:55:45,149 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:55:45,149 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testSplitTableRegionEmptyDaughter' starting 'acquire' 2023-04-12 02:55:45,149 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testSplitTableRegionEmptyDaughter', kicking off acquire phase on members. 2023-04-12 02:55:45,149 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:45,149 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:45,150 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:45,150 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:55:45,150 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:55:45,150 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:55:45,151 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:55:45,151 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:55:45,151 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:45,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:55:45,151 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:55:45,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:55:45,151 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 02:55:45,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:55:45,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:45,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:45,152 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:45,152 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:45,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 02:55:45,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:45,152 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionEmptyDaughter 2023-04-12 02:55:45,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 02:55:45,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:45,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:45,152 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionEmptyDaughter 2023-04-12 02:55:45,152 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionEmptyDaughter is still present, removing 2023-04-12 02:55:45,152 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionEmptyDaughter 2023-04-12 02:55:45,152 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionEmptyDaughter is still present, removing 2023-04-12 02:55:45,152 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionEmptyDaughter' with timeout 60000ms 2023-04-12 02:55:45,153 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:55:45,153 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionEmptyDaughter' starting 'acquire' stage 2023-04-12 02:55:45,153 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionEmptyDaughter 2023-04-12 02:55:45,153 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionEmptyDaughter' locally acquired 2023-04-12 02:55:45,153 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionEmptyDaughter' with timeout 60000ms 2023-04-12 02:55:45,153 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:55:45,153 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44679,1681268031772' joining acquired barrier for procedure (testSplitTableRegionEmptyDaughter) in zk 2023-04-12 02:55:45,153 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:45,153 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionEmptyDaughter' starting 'acquire' stage 2023-04-12 02:55:45,153 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 02:55:45,153 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 02:55:45,153 DEBUG [rs(jenkins-hbase4.apache.org,43091,1681268031804)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testSplitTableRegionEmptyDaughter,,1681268143910.7e32820b75c2980cc8d414d4b96996d4. 2023-04-12 02:55:45,153 DEBUG [rs(jenkins-hbase4.apache.org,43091,1681268031804)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testSplitTableRegionEmptyDaughter,,1681268143910.7e32820b75c2980cc8d414d4b96996d4. started... 2023-04-12 02:55:45,153 INFO [rs(jenkins-hbase4.apache.org,43091,1681268031804)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 7e32820b75c2980cc8d414d4b96996d4 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-04-12 02:55:45,156 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:45,156 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:45,156 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionEmptyDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 02:55:45,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 02:55:45,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:45,160 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionEmptyDaughter 2023-04-12 02:55:45,172 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionEmptyDaughter is still present, removing 2023-04-12 02:55:45,172 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionEmptyDaughter 2023-04-12 02:55:45,172 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionEmptyDaughter' with timeout 60000ms 2023-04-12 02:55:45,172 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:55:45,172 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionEmptyDaughter' starting 'acquire' stage 2023-04-12 02:55:45,172 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionEmptyDaughter' locally acquired 2023-04-12 02:55:45,172 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34579,1681268031719' joining acquired barrier for procedure (testSplitTableRegionEmptyDaughter) in zk 2023-04-12 02:55:45,174 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:45,174 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:45,174 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionEmptyDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 02:55:45,567 INFO [rs(jenkins-hbase4.apache.org,43091,1681268031804)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=19 (bloomFilter=true), to=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/7e32820b75c2980cc8d414d4b96996d4/.tmp/cf1/db8da92f5ccc4286bc7911a7912d53db 2023-04-12 02:55:45,578 INFO [rs(jenkins-hbase4.apache.org,43091,1681268031804)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=19 (bloomFilter=true), to=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/7e32820b75c2980cc8d414d4b96996d4/.tmp/cf2/5e06069d9e7543b390df83212345aa28 2023-04-12 02:55:45,582 DEBUG [rs(jenkins-hbase4.apache.org,43091,1681268031804)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/7e32820b75c2980cc8d414d4b96996d4/.tmp/cf1/db8da92f5ccc4286bc7911a7912d53db as hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/7e32820b75c2980cc8d414d4b96996d4/cf1/db8da92f5ccc4286bc7911a7912d53db 2023-04-12 02:55:45,586 INFO [rs(jenkins-hbase4.apache.org,43091,1681268031804)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/7e32820b75c2980cc8d414d4b96996d4/cf1/db8da92f5ccc4286bc7911a7912d53db, entries=10, sequenceid=19, filesize=5.0 K 2023-04-12 02:55:45,587 DEBUG [rs(jenkins-hbase4.apache.org,43091,1681268031804)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/7e32820b75c2980cc8d414d4b96996d4/.tmp/cf2/5e06069d9e7543b390df83212345aa28 as hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/7e32820b75c2980cc8d414d4b96996d4/cf2/5e06069d9e7543b390df83212345aa28 2023-04-12 02:55:45,590 INFO [rs(jenkins-hbase4.apache.org,43091,1681268031804)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/7e32820b75c2980cc8d414d4b96996d4/cf2/5e06069d9e7543b390df83212345aa28, entries=10, sequenceid=19, filesize=5.0 K 2023-04-12 02:55:45,593 INFO [rs(jenkins-hbase4.apache.org,43091,1681268031804)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for 7e32820b75c2980cc8d414d4b96996d4 in 439ms, sequenceid=19, compaction requested=false 2023-04-12 02:55:45,593 DEBUG [rs(jenkins-hbase4.apache.org,43091,1681268031804)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7e32820b75c2980cc8d414d4b96996d4: 2023-04-12 02:55:45,593 DEBUG [rs(jenkins-hbase4.apache.org,43091,1681268031804)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testSplitTableRegionEmptyDaughter,,1681268143910.7e32820b75c2980cc8d414d4b96996d4. 2023-04-12 02:55:45,593 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 02:55:45,593 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 02:55:45,593 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:45,593 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionEmptyDaughter' locally acquired 2023-04-12 02:55:45,593 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43091,1681268031804' joining acquired barrier for procedure (testSplitTableRegionEmptyDaughter) in zk 2023-04-12 02:55:45,595 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:45,595 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:45,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:45,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 02:55:45,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 02:55:45,595 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:45,595 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionEmptyDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 02:55:45,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 02:55:45,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 02:55:45,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionEmptyDaughter 2023-04-12 02:55:45,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:45,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:45,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:45,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 02:55:45,597 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43091,1681268031804' joining acquired barrier for procedure 'testSplitTableRegionEmptyDaughter' on coordinator 2023-04-12 02:55:45,597 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@32ed9660[Count = 0] remaining members to acquire global barrier 2023-04-12 02:55:45,598 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testSplitTableRegionEmptyDaughter' starting 'in-barrier' execution. 2023-04-12 02:55:45,598 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:45,599 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:45,599 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:45,599 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:45,599 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:45,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:45,599 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:45,599 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionEmptyDaughter' received 'reached' from coordinator. 2023-04-12 02:55:45,599 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionEmptyDaughter' locally completed 2023-04-12 02:55:45,599 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:45,599 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 02:55:45,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:45,599 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:45,599 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionEmptyDaughter' received 'reached' from coordinator. 2023-04-12 02:55:45,599 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionEmptyDaughter' locally completed 2023-04-12 02:55:45,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:45,599 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionEmptyDaughter' completed for member 'jenkins-hbase4.apache.org,43091,1681268031804' in zk 2023-04-12 02:55:45,599 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionEmptyDaughter' received 'reached' from coordinator. 2023-04-12 02:55:45,599 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionEmptyDaughter' locally completed 2023-04-12 02:55:45,599 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionEmptyDaughter' completed for member 'jenkins-hbase4.apache.org,34579,1681268031719' in zk 2023-04-12 02:55:45,599 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionEmptyDaughter' completed for member 'jenkins-hbase4.apache.org,44679,1681268031772' in zk 2023-04-12 02:55:45,602 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionEmptyDaughter' has notified controller of completion 2023-04-12 02:55:45,602 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:45,602 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionEmptyDaughter' has notified controller of completion 2023-04-12 02:55:45,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:45,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 02:55:45,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 02:55:45,602 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionEmptyDaughter' has notified controller of completion 2023-04-12 02:55:45,602 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:55:45,602 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:55:45,603 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionEmptyDaughter' completed. 2023-04-12 02:55:45,603 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionEmptyDaughter' completed. 2023-04-12 02:55:45,602 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:55:45,603 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionEmptyDaughter' completed. 2023-04-12 02:55:45,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 02:55:45,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 02:55:45,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionEmptyDaughter 2023-04-12 02:55:45,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:45,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:45,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:45,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 02:55:45,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionEmptyDaughter 2023-04-12 02:55:45,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:45,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:45,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:45,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testSplitTableRegionEmptyDaughter' member 'jenkins-hbase4.apache.org,43091,1681268031804': 2023-04-12 02:55:45,606 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43091,1681268031804' released barrier for procedure'testSplitTableRegionEmptyDaughter', counting down latch. Waiting for 0 more 2023-04-12 02:55:45,606 INFO [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testSplitTableRegionEmptyDaughter' execution completed 2023-04-12 02:55:45,606 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 02:55:45,606 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 02:55:45,606 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testSplitTableRegionEmptyDaughter 2023-04-12 02:55:45,606 INFO [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testSplitTableRegionEmptyDaughterincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 02:55:45,608 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:45,608 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:55:45,608 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:45,608 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:45,608 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:45,608 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:55:45,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:45,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 02:55:45,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 02:55:45,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:45,608 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:45,609 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:55:45,608 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:45,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:45,609 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:45,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:45,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 02:55:45,609 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:45,609 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:55:45,609 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:55:45,609 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:55:45,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:55:45,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:55:45,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:55:45,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionEmptyDaughter 2023-04-12 02:55:45,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:45,610 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:45,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:45,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 02:55:45,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:45,611 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:45,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionEmptyDaughter 2023-04-12 02:55:45,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:45,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:45,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:45,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 02:55:45,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionEmptyDaughter 2023-04-12 02:55:45,612 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:45,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:45,613 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:45,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:45,613 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:45,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:45,616 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:55:45,616 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:55:45,616 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:55:45,616 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:45,616 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testSplitTableRegionEmptyDaughter' 2023-04-12 02:55:45,616 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 02:55:45,616 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:45,616 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:45,616 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:45,616 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:55:45,616 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:45,616 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:45,616 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:45,616 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:45,616 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:45,616 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:45,616 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:45,616 DEBUG [Listener at localhost/46667] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testSplitTableRegionEmptyDaughter'' to complete. (max 20000 ms per retry) 2023-04-12 02:55:45,616 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:55:45,617 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:55:45,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:55:45,617 DEBUG [Listener at localhost/46667] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 02:55:45,617 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:55:45,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:55:45,617 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:55:45,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:55:45,617 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:55:45,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:55:45,620 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:55:45,620 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:55:45,620 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:55:45,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:55:45,620 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:55:45,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:55:45,717 DEBUG [Listener at localhost/46667] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 02:55:45,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 02:55:45,731 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 02:55:45,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] procedure.ProcedureCoordinator(143): Procedure testSplitTableRegionEmptyDaughter was in running list but was completed. Accepting new attempt. 2023-04-12 02:55:45,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] procedure.ProcedureCoordinator(165): Submitting procedure testSplitTableRegionEmptyDaughter 2023-04-12 02:55:45,733 INFO [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testSplitTableRegionEmptyDaughter' 2023-04-12 02:55:45,733 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:55:45,733 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testSplitTableRegionEmptyDaughter' starting 'acquire' 2023-04-12 02:55:45,733 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testSplitTableRegionEmptyDaughter', kicking off acquire phase on members. 2023-04-12 02:55:45,733 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:45,734 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:45,737 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:55:45,737 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:45,737 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:55:45,738 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:55:45,738 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:55:45,738 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:55:45,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:55:45,738 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:55:45,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:55:45,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:55:45,738 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:45,738 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 02:55:45,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:45,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:45,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:45,739 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:45,739 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:45,739 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:45,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 02:55:45,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:45,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 02:55:45,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:45,739 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionEmptyDaughter 2023-04-12 02:55:45,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 02:55:45,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:45,739 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionEmptyDaughter 2023-04-12 02:55:45,739 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionEmptyDaughter 2023-04-12 02:55:45,740 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionEmptyDaughter 2023-04-12 02:55:45,740 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionEmptyDaughter 2023-04-12 02:55:45,740 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionEmptyDaughter 2023-04-12 02:55:45,740 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionEmptyDaughter' with timeout 60000ms 2023-04-12 02:55:45,740 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionEmptyDaughter' with timeout 60000ms 2023-04-12 02:55:45,740 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionEmptyDaughter' with timeout 60000ms 2023-04-12 02:55:45,740 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:55:45,740 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:55:45,740 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:55:45,740 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionEmptyDaughter' starting 'acquire' stage 2023-04-12 02:55:45,740 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionEmptyDaughter' starting 'acquire' stage 2023-04-12 02:55:45,740 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionEmptyDaughter' starting 'acquire' stage 2023-04-12 02:55:45,740 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionEmptyDaughter' locally acquired 2023-04-12 02:55:45,740 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionEmptyDaughter' locally acquired 2023-04-12 02:55:45,740 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34579,1681268031719' joining acquired barrier for procedure (testSplitTableRegionEmptyDaughter) in zk 2023-04-12 02:55:45,740 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44679,1681268031772' joining acquired barrier for procedure (testSplitTableRegionEmptyDaughter) in zk 2023-04-12 02:55:45,740 DEBUG [rs(jenkins-hbase4.apache.org,43091,1681268031804)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testSplitTableRegionEmptyDaughter,,1681268143910.7e32820b75c2980cc8d414d4b96996d4. 2023-04-12 02:55:45,740 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 02:55:45,740 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 02:55:45,740 DEBUG [rs(jenkins-hbase4.apache.org,43091,1681268031804)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testSplitTableRegionEmptyDaughter,,1681268143910.7e32820b75c2980cc8d414d4b96996d4. started... 2023-04-12 02:55:45,741 INFO [rs(jenkins-hbase4.apache.org,43091,1681268031804)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 7e32820b75c2980cc8d414d4b96996d4 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-04-12 02:55:45,744 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:45,747 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:45,747 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:45,747 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionEmptyDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 02:55:45,747 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:45,747 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionEmptyDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 02:55:45,757 INFO [rs(jenkins-hbase4.apache.org,43091,1681268031804)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=32 (bloomFilter=true), to=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/7e32820b75c2980cc8d414d4b96996d4/.tmp/cf1/eb05d01415cc4933b6888f334b476175 2023-04-12 02:55:45,784 INFO [rs(jenkins-hbase4.apache.org,43091,1681268031804)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=32 (bloomFilter=true), to=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/7e32820b75c2980cc8d414d4b96996d4/.tmp/cf2/3f2612ae7a8b4c729ad90a32b95b9ce8 2023-04-12 02:55:45,788 DEBUG [rs(jenkins-hbase4.apache.org,43091,1681268031804)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/7e32820b75c2980cc8d414d4b96996d4/.tmp/cf1/eb05d01415cc4933b6888f334b476175 as hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/7e32820b75c2980cc8d414d4b96996d4/cf1/eb05d01415cc4933b6888f334b476175 2023-04-12 02:55:45,793 INFO [rs(jenkins-hbase4.apache.org,43091,1681268031804)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/7e32820b75c2980cc8d414d4b96996d4/cf1/eb05d01415cc4933b6888f334b476175, entries=10, sequenceid=32, filesize=5.0 K 2023-04-12 02:55:45,793 DEBUG [rs(jenkins-hbase4.apache.org,43091,1681268031804)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/7e32820b75c2980cc8d414d4b96996d4/.tmp/cf2/3f2612ae7a8b4c729ad90a32b95b9ce8 as hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/7e32820b75c2980cc8d414d4b96996d4/cf2/3f2612ae7a8b4c729ad90a32b95b9ce8 2023-04-12 02:55:45,797 INFO [rs(jenkins-hbase4.apache.org,43091,1681268031804)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/7e32820b75c2980cc8d414d4b96996d4/cf2/3f2612ae7a8b4c729ad90a32b95b9ce8, entries=10, sequenceid=32, filesize=5.0 K 2023-04-12 02:55:45,798 INFO [rs(jenkins-hbase4.apache.org,43091,1681268031804)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for 7e32820b75c2980cc8d414d4b96996d4 in 57ms, sequenceid=32, compaction requested=true 2023-04-12 02:55:45,799 DEBUG [rs(jenkins-hbase4.apache.org,43091,1681268031804)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7e32820b75c2980cc8d414d4b96996d4: 2023-04-12 02:55:45,799 DEBUG [rs(jenkins-hbase4.apache.org,43091,1681268031804)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testSplitTableRegionEmptyDaughter,,1681268143910.7e32820b75c2980cc8d414d4b96996d4. 2023-04-12 02:55:45,799 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 02:55:45,799 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 02:55:45,799 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:45,799 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionEmptyDaughter' locally acquired 2023-04-12 02:55:45,799 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43091,1681268031804' joining acquired barrier for procedure (testSplitTableRegionEmptyDaughter) in zk 2023-04-12 02:55:45,801 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:45,801 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:45,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:45,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 02:55:45,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 02:55:45,802 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:45,802 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionEmptyDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 02:55:45,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 02:55:45,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 02:55:45,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionEmptyDaughter 2023-04-12 02:55:45,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:45,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:45,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:45,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 02:55:45,804 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43091,1681268031804' joining acquired barrier for procedure 'testSplitTableRegionEmptyDaughter' on coordinator 2023-04-12 02:55:45,804 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1ea1a58e[Count = 0] remaining members to acquire global barrier 2023-04-12 02:55:45,804 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testSplitTableRegionEmptyDaughter' starting 'in-barrier' execution. 2023-04-12 02:55:45,804 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:45,805 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:45,805 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:45,805 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:45,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:45,805 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:45,806 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionEmptyDaughter' received 'reached' from coordinator. 2023-04-12 02:55:45,806 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:45,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:45,805 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:45,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:45,806 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionEmptyDaughter' received 'reached' from coordinator. 2023-04-12 02:55:45,806 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:45,806 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 02:55:45,806 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionEmptyDaughter' locally completed 2023-04-12 02:55:45,806 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionEmptyDaughter' locally completed 2023-04-12 02:55:45,806 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionEmptyDaughter' completed for member 'jenkins-hbase4.apache.org,44679,1681268031772' in zk 2023-04-12 02:55:45,806 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionEmptyDaughter' received 'reached' from coordinator. 2023-04-12 02:55:45,806 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionEmptyDaughter' locally completed 2023-04-12 02:55:45,806 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionEmptyDaughter' completed for member 'jenkins-hbase4.apache.org,34579,1681268031719' in zk 2023-04-12 02:55:45,806 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionEmptyDaughter' completed for member 'jenkins-hbase4.apache.org,43091,1681268031804' in zk 2023-04-12 02:55:45,815 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionEmptyDaughter' has notified controller of completion 2023-04-12 02:55:45,815 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionEmptyDaughter' has notified controller of completion 2023-04-12 02:55:45,815 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:55:45,815 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:55:45,815 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionEmptyDaughter' completed. 2023-04-12 02:55:45,816 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionEmptyDaughter' completed. 2023-04-12 02:55:45,816 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionEmptyDaughter' has notified controller of completion 2023-04-12 02:55:45,816 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:55:45,816 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:45,817 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionEmptyDaughter' completed. 2023-04-12 02:55:45,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:45,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 02:55:45,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 02:55:45,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 02:55:45,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 02:55:45,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionEmptyDaughter 2023-04-12 02:55:45,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:45,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:45,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:45,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 02:55:45,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionEmptyDaughter 2023-04-12 02:55:45,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:45,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:45,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:45,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testSplitTableRegionEmptyDaughter' member 'jenkins-hbase4.apache.org,43091,1681268031804': 2023-04-12 02:55:45,821 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43091,1681268031804' released barrier for procedure'testSplitTableRegionEmptyDaughter', counting down latch. Waiting for 0 more 2023-04-12 02:55:45,821 INFO [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testSplitTableRegionEmptyDaughter' execution completed 2023-04-12 02:55:45,821 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 02:55:45,821 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 02:55:45,821 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testSplitTableRegionEmptyDaughter 2023-04-12 02:55:45,821 INFO [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testSplitTableRegionEmptyDaughterincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 02:55:45,824 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:45,824 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:55:45,824 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:45,824 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:45,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:45,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 02:55:45,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 02:55:45,824 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:45,824 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:55:45,824 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:45,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:45,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:45,824 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:45,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:55:45,825 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:45,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:55:45,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:55:45,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:55:45,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:45,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:45,825 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:55:45,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 02:55:45,825 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:45,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:45,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:45,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:55:45,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:55:45,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionEmptyDaughter 2023-04-12 02:55:45,826 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:45,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:45,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 02:55:45,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionEmptyDaughter 2023-04-12 02:55:45,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:45,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:45,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:45,827 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:45,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 02:55:45,827 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:45,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionEmptyDaughter 2023-04-12 02:55:45,828 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:45,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:45,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:45,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:45,831 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:55:45,831 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:55:45,831 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:55:45,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:55:45,831 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:55:45,831 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:55:45,831 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:55:45,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:55:45,831 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:45,831 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:45,831 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:45,831 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:45,831 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:45,831 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:45,831 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:45,831 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:45,831 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:45,831 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:55:45,831 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:55:45,831 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:55:45,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:55:45,831 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:55:45,832 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:55:45,831 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testSplitTableRegionEmptyDaughter' 2023-04-12 02:55:45,832 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 02:55:45,831 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:45,832 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:55:45,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:55:45,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:55:45,832 DEBUG [Listener at localhost/46667] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testSplitTableRegionEmptyDaughter'' to complete. (max 20000 ms per retry) 2023-04-12 02:55:45,832 DEBUG [Listener at localhost/46667] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 02:55:45,832 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:45,832 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:55:45,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:55:45,932 DEBUG [Listener at localhost/46667] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 02:55:45,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 02:55:45,951 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 02:55:45,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] procedure.ProcedureCoordinator(143): Procedure testSplitTableRegionEmptyDaughter was in running list but was completed. Accepting new attempt. 2023-04-12 02:55:45,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] procedure.ProcedureCoordinator(165): Submitting procedure testSplitTableRegionEmptyDaughter 2023-04-12 02:55:45,953 INFO [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testSplitTableRegionEmptyDaughter' 2023-04-12 02:55:45,953 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:55:45,953 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testSplitTableRegionEmptyDaughter' starting 'acquire' 2023-04-12 02:55:45,953 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testSplitTableRegionEmptyDaughter', kicking off acquire phase on members. 2023-04-12 02:55:45,954 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:45,954 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:45,956 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:45,956 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:55:45,956 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:55:45,956 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:55:45,956 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:55:45,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:55:45,956 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:55:45,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:55:45,956 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:45,956 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 02:55:45,956 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:55:45,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:55:45,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:45,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:45,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:45,957 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:45,957 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:45,958 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:45,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 02:55:45,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:45,958 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionEmptyDaughter 2023-04-12 02:55:45,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 02:55:45,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:45,958 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionEmptyDaughter 2023-04-12 02:55:45,958 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionEmptyDaughter is still present, removing 2023-04-12 02:55:45,958 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionEmptyDaughter 2023-04-12 02:55:45,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 02:55:45,958 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionEmptyDaughter' with timeout 60000ms 2023-04-12 02:55:45,958 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:55:45,958 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionEmptyDaughter' starting 'acquire' stage 2023-04-12 02:55:45,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:45,958 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionEmptyDaughter 2023-04-12 02:55:45,964 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionEmptyDaughter is still present, removing 2023-04-12 02:55:45,964 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionEmptyDaughter 2023-04-12 02:55:45,964 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionEmptyDaughter' with timeout 60000ms 2023-04-12 02:55:45,964 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:55:45,964 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionEmptyDaughter is still present, removing 2023-04-12 02:55:45,964 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionEmptyDaughter' starting 'acquire' stage 2023-04-12 02:55:45,964 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 02:55:45,964 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionEmptyDaughter' locally acquired 2023-04-12 02:55:45,964 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionEmptyDaughter 2023-04-12 02:55:45,964 DEBUG [rs(jenkins-hbase4.apache.org,43091,1681268031804)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testSplitTableRegionEmptyDaughter,,1681268143910.7e32820b75c2980cc8d414d4b96996d4. 2023-04-12 02:55:45,964 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionEmptyDaughter' with timeout 60000ms 2023-04-12 02:55:45,964 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:55:45,964 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34579,1681268031719' joining acquired barrier for procedure (testSplitTableRegionEmptyDaughter) in zk 2023-04-12 02:55:45,964 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionEmptyDaughter' starting 'acquire' stage 2023-04-12 02:55:45,964 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionEmptyDaughter' locally acquired 2023-04-12 02:55:45,964 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44679,1681268031772' joining acquired barrier for procedure (testSplitTableRegionEmptyDaughter) in zk 2023-04-12 02:55:45,964 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 02:55:45,964 DEBUG [rs(jenkins-hbase4.apache.org,43091,1681268031804)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testSplitTableRegionEmptyDaughter,,1681268143910.7e32820b75c2980cc8d414d4b96996d4. started... 2023-04-12 02:55:45,965 INFO [rs(jenkins-hbase4.apache.org,43091,1681268031804)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 7e32820b75c2980cc8d414d4b96996d4 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-04-12 02:55:45,966 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:45,967 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:45,967 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:45,968 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionEmptyDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 02:55:45,968 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:45,968 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionEmptyDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 02:55:45,976 INFO [rs(jenkins-hbase4.apache.org,43091,1681268031804)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=45 (bloomFilter=true), to=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/7e32820b75c2980cc8d414d4b96996d4/.tmp/cf1/e70af74642cc4f69ade829953a2cb4e3 2023-04-12 02:55:45,988 INFO [rs(jenkins-hbase4.apache.org,43091,1681268031804)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=45 (bloomFilter=true), to=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/7e32820b75c2980cc8d414d4b96996d4/.tmp/cf2/ed471e0ca1f449d7b18c5368407dcfa0 2023-04-12 02:55:45,993 DEBUG [rs(jenkins-hbase4.apache.org,43091,1681268031804)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/7e32820b75c2980cc8d414d4b96996d4/.tmp/cf1/e70af74642cc4f69ade829953a2cb4e3 as hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/7e32820b75c2980cc8d414d4b96996d4/cf1/e70af74642cc4f69ade829953a2cb4e3 2023-04-12 02:55:45,997 INFO [rs(jenkins-hbase4.apache.org,43091,1681268031804)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/7e32820b75c2980cc8d414d4b96996d4/cf1/e70af74642cc4f69ade829953a2cb4e3, entries=10, sequenceid=45, filesize=5.0 K 2023-04-12 02:55:45,998 DEBUG [rs(jenkins-hbase4.apache.org,43091,1681268031804)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/7e32820b75c2980cc8d414d4b96996d4/.tmp/cf2/ed471e0ca1f449d7b18c5368407dcfa0 as hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/7e32820b75c2980cc8d414d4b96996d4/cf2/ed471e0ca1f449d7b18c5368407dcfa0 2023-04-12 02:55:46,001 INFO [rs(jenkins-hbase4.apache.org,43091,1681268031804)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/7e32820b75c2980cc8d414d4b96996d4/cf2/ed471e0ca1f449d7b18c5368407dcfa0, entries=10, sequenceid=45, filesize=5.0 K 2023-04-12 02:55:46,002 INFO [rs(jenkins-hbase4.apache.org,43091,1681268031804)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for 7e32820b75c2980cc8d414d4b96996d4 in 37ms, sequenceid=45, compaction requested=true 2023-04-12 02:55:46,002 DEBUG [rs(jenkins-hbase4.apache.org,43091,1681268031804)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7e32820b75c2980cc8d414d4b96996d4: 2023-04-12 02:55:46,002 DEBUG [rs(jenkins-hbase4.apache.org,43091,1681268031804)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testSplitTableRegionEmptyDaughter,,1681268143910.7e32820b75c2980cc8d414d4b96996d4. 2023-04-12 02:55:46,002 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 02:55:46,002 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 02:55:46,002 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:46,002 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionEmptyDaughter' locally acquired 2023-04-12 02:55:46,002 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43091,1681268031804' joining acquired barrier for procedure (testSplitTableRegionEmptyDaughter) in zk 2023-04-12 02:55:46,004 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:46,004 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:46,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:46,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 02:55:46,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 02:55:46,004 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:46,004 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionEmptyDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 02:55:46,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 02:55:46,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 02:55:46,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionEmptyDaughter 2023-04-12 02:55:46,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:46,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:46,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:46,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 02:55:46,007 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43091,1681268031804' joining acquired barrier for procedure 'testSplitTableRegionEmptyDaughter' on coordinator 2023-04-12 02:55:46,007 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7628634[Count = 0] remaining members to acquire global barrier 2023-04-12 02:55:46,007 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testSplitTableRegionEmptyDaughter' starting 'in-barrier' execution. 2023-04-12 02:55:46,007 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:46,009 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:46,009 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:46,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:46,009 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionEmptyDaughter' received 'reached' from coordinator. 2023-04-12 02:55:46,009 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionEmptyDaughter' locally completed 2023-04-12 02:55:46,009 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionEmptyDaughter' completed for member 'jenkins-hbase4.apache.org,44679,1681268031772' in zk 2023-04-12 02:55:46,009 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:46,009 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:46,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:46,009 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:46,009 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:46,009 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionEmptyDaughter' received 'reached' from coordinator. 2023-04-12 02:55:46,010 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionEmptyDaughter' locally completed 2023-04-12 02:55:46,010 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionEmptyDaughter' completed for member 'jenkins-hbase4.apache.org,34579,1681268031719' in zk 2023-04-12 02:55:46,009 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:46,010 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 02:55:46,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:46,010 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionEmptyDaughter' received 'reached' from coordinator. 2023-04-12 02:55:46,010 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionEmptyDaughter' locally completed 2023-04-12 02:55:46,010 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionEmptyDaughter' completed for member 'jenkins-hbase4.apache.org,43091,1681268031804' in zk 2023-04-12 02:55:46,011 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionEmptyDaughter' has notified controller of completion 2023-04-12 02:55:46,011 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:55:46,011 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionEmptyDaughter' completed. 2023-04-12 02:55:46,012 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:46,012 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionEmptyDaughter' has notified controller of completion 2023-04-12 02:55:46,012 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:55:46,012 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionEmptyDaughter' has notified controller of completion 2023-04-12 02:55:46,012 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionEmptyDaughter' completed. 2023-04-12 02:55:46,012 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:55:46,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:46,012 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionEmptyDaughter' completed. 2023-04-12 02:55:46,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 02:55:46,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 02:55:46,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 02:55:46,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 02:55:46,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionEmptyDaughter 2023-04-12 02:55:46,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:46,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:46,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:46,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 02:55:46,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionEmptyDaughter 2023-04-12 02:55:46,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:46,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:46,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:46,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testSplitTableRegionEmptyDaughter' member 'jenkins-hbase4.apache.org,43091,1681268031804': 2023-04-12 02:55:46,016 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43091,1681268031804' released barrier for procedure'testSplitTableRegionEmptyDaughter', counting down latch. Waiting for 0 more 2023-04-12 02:55:46,016 INFO [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testSplitTableRegionEmptyDaughter' execution completed 2023-04-12 02:55:46,016 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 02:55:46,016 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 02:55:46,017 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testSplitTableRegionEmptyDaughter 2023-04-12 02:55:46,017 INFO [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testSplitTableRegionEmptyDaughterincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 02:55:46,018 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:46,018 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:46,018 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:46,018 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:46,018 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:46,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:46,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:46,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 02:55:46,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 02:55:46,018 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:55:46,018 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:55:46,018 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:46,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:46,019 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:55:46,019 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:46,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:46,021 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:55:46,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:55:46,021 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:55:46,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:55:46,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 02:55:46,021 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:55:46,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:55:46,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:46,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:46,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:46,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionEmptyDaughter 2023-04-12 02:55:46,022 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:46,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 02:55:46,022 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:46,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionEmptyDaughter 2023-04-12 02:55:46,022 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:46,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:46,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:46,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:46,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 02:55:46,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionEmptyDaughter 2023-04-12 02:55:46,024 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:46,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:46,024 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:46,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:46,024 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:46,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:46,034 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:46,034 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:46,034 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:46,035 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:55:46,035 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:46,035 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:46,035 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:55:46,035 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:55:46,035 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testSplitTableRegionEmptyDaughter' 2023-04-12 02:55:46,035 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 02:55:46,035 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:46,035 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:55:46,035 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:55:46,035 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:55:46,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:55:46,035 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:55:46,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:55:46,035 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:46,035 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:55:46,035 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:55:46,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:55:46,035 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:46,035 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:46,035 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:46,036 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:46,035 DEBUG [Listener at localhost/46667] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testSplitTableRegionEmptyDaughter'' to complete. (max 20000 ms per retry) 2023-04-12 02:55:46,035 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:55:46,036 DEBUG [Listener at localhost/46667] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 02:55:46,036 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:55:46,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:55:46,035 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:55:46,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:55:46,036 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:55:46,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:55:46,136 DEBUG [Listener at localhost/46667] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 02:55:46,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 02:55:46,149 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 02:55:46,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] procedure.ProcedureCoordinator(143): Procedure testSplitTableRegionEmptyDaughter was in running list but was completed. Accepting new attempt. 2023-04-12 02:55:46,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] procedure.ProcedureCoordinator(165): Submitting procedure testSplitTableRegionEmptyDaughter 2023-04-12 02:55:46,150 INFO [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testSplitTableRegionEmptyDaughter' 2023-04-12 02:55:46,150 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:55:46,151 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testSplitTableRegionEmptyDaughter' starting 'acquire' 2023-04-12 02:55:46,151 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testSplitTableRegionEmptyDaughter', kicking off acquire phase on members. 2023-04-12 02:55:46,151 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:46,151 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:46,152 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:55:46,152 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:46,152 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:55:46,152 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:55:46,152 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:55:46,152 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:55:46,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:55:46,153 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:55:46,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:55:46,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:55:46,153 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:46,153 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 02:55:46,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:46,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:46,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:46,153 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:46,153 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:46,153 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:46,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 02:55:46,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:46,154 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionEmptyDaughter 2023-04-12 02:55:46,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 02:55:46,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:46,154 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionEmptyDaughter 2023-04-12 02:55:46,154 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionEmptyDaughter is still present, removing 2023-04-12 02:55:46,154 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionEmptyDaughter 2023-04-12 02:55:46,154 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionEmptyDaughter' with timeout 60000ms 2023-04-12 02:55:46,154 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:55:46,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 02:55:46,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:46,154 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionEmptyDaughter' starting 'acquire' stage 2023-04-12 02:55:46,154 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionEmptyDaughter 2023-04-12 02:55:46,154 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 02:55:46,155 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 02:55:46,154 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionEmptyDaughter is still present, removing 2023-04-12 02:55:46,155 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionEmptyDaughter 2023-04-12 02:55:46,155 DEBUG [rs(jenkins-hbase4.apache.org,43091,1681268031804)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testSplitTableRegionEmptyDaughter,,1681268143910.7e32820b75c2980cc8d414d4b96996d4. 2023-04-12 02:55:46,155 DEBUG [rs(jenkins-hbase4.apache.org,43091,1681268031804)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testSplitTableRegionEmptyDaughter,,1681268143910.7e32820b75c2980cc8d414d4b96996d4. started... 2023-04-12 02:55:46,155 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionEmptyDaughter is still present, removing 2023-04-12 02:55:46,155 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionEmptyDaughter' with timeout 60000ms 2023-04-12 02:55:46,155 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:55:46,155 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionEmptyDaughter 2023-04-12 02:55:46,155 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionEmptyDaughter' starting 'acquire' stage 2023-04-12 02:55:46,155 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionEmptyDaughter' locally acquired 2023-04-12 02:55:46,155 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionEmptyDaughter' with timeout 60000ms 2023-04-12 02:55:46,155 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:55:46,155 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44679,1681268031772' joining acquired barrier for procedure (testSplitTableRegionEmptyDaughter) in zk 2023-04-12 02:55:46,155 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionEmptyDaughter' starting 'acquire' stage 2023-04-12 02:55:46,155 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionEmptyDaughter' locally acquired 2023-04-12 02:55:46,155 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34579,1681268031719' joining acquired barrier for procedure (testSplitTableRegionEmptyDaughter) in zk 2023-04-12 02:55:46,156 INFO [rs(jenkins-hbase4.apache.org,43091,1681268031804)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 7e32820b75c2980cc8d414d4b96996d4 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-04-12 02:55:46,163 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:46,163 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:46,163 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:46,163 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionEmptyDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 02:55:46,163 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:46,163 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionEmptyDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 02:55:46,170 INFO [rs(jenkins-hbase4.apache.org,43091,1681268031804)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=58 (bloomFilter=true), to=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/7e32820b75c2980cc8d414d4b96996d4/.tmp/cf1/0039c871981747f0a784fb0e99655f62 2023-04-12 02:55:46,186 INFO [rs(jenkins-hbase4.apache.org,43091,1681268031804)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=58 (bloomFilter=true), to=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/7e32820b75c2980cc8d414d4b96996d4/.tmp/cf2/ff655bed4b6d4c67a983029ee068984b 2023-04-12 02:55:46,189 DEBUG [rs(jenkins-hbase4.apache.org,43091,1681268031804)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/7e32820b75c2980cc8d414d4b96996d4/.tmp/cf1/0039c871981747f0a784fb0e99655f62 as hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/7e32820b75c2980cc8d414d4b96996d4/cf1/0039c871981747f0a784fb0e99655f62 2023-04-12 02:55:46,193 INFO [rs(jenkins-hbase4.apache.org,43091,1681268031804)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/7e32820b75c2980cc8d414d4b96996d4/cf1/0039c871981747f0a784fb0e99655f62, entries=10, sequenceid=58, filesize=5.0 K 2023-04-12 02:55:46,193 DEBUG [rs(jenkins-hbase4.apache.org,43091,1681268031804)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/7e32820b75c2980cc8d414d4b96996d4/.tmp/cf2/ff655bed4b6d4c67a983029ee068984b as hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/7e32820b75c2980cc8d414d4b96996d4/cf2/ff655bed4b6d4c67a983029ee068984b 2023-04-12 02:55:46,197 INFO [rs(jenkins-hbase4.apache.org,43091,1681268031804)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/7e32820b75c2980cc8d414d4b96996d4/cf2/ff655bed4b6d4c67a983029ee068984b, entries=10, sequenceid=58, filesize=5.0 K 2023-04-12 02:55:46,198 INFO [rs(jenkins-hbase4.apache.org,43091,1681268031804)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for 7e32820b75c2980cc8d414d4b96996d4 in 42ms, sequenceid=58, compaction requested=true 2023-04-12 02:55:46,198 DEBUG [rs(jenkins-hbase4.apache.org,43091,1681268031804)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7e32820b75c2980cc8d414d4b96996d4: 2023-04-12 02:55:46,198 DEBUG [rs(jenkins-hbase4.apache.org,43091,1681268031804)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testSplitTableRegionEmptyDaughter,,1681268143910.7e32820b75c2980cc8d414d4b96996d4. 2023-04-12 02:55:46,198 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 02:55:46,198 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 02:55:46,198 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:46,198 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionEmptyDaughter' locally acquired 2023-04-12 02:55:46,198 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43091,1681268031804' joining acquired barrier for procedure (testSplitTableRegionEmptyDaughter) in zk 2023-04-12 02:55:46,199 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:46,199 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:46,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:46,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 02:55:46,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 02:55:46,200 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:46,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 02:55:46,200 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionEmptyDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 02:55:46,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 02:55:46,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionEmptyDaughter 2023-04-12 02:55:46,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:46,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:46,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:46,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 02:55:46,202 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43091,1681268031804' joining acquired barrier for procedure 'testSplitTableRegionEmptyDaughter' on coordinator 2023-04-12 02:55:46,202 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@34d7f4bc[Count = 0] remaining members to acquire global barrier 2023-04-12 02:55:46,202 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testSplitTableRegionEmptyDaughter' starting 'in-barrier' execution. 2023-04-12 02:55:46,202 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:46,203 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:46,203 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:46,203 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:46,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:46,203 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionEmptyDaughter' received 'reached' from coordinator. 2023-04-12 02:55:46,203 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:46,203 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 02:55:46,203 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:46,203 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionEmptyDaughter' locally completed 2023-04-12 02:55:46,203 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionEmptyDaughter' completed for member 'jenkins-hbase4.apache.org,44679,1681268031772' in zk 2023-04-12 02:55:46,203 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:46,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:46,203 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:46,203 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionEmptyDaughter' received 'reached' from coordinator. 2023-04-12 02:55:46,203 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionEmptyDaughter' locally completed 2023-04-12 02:55:46,203 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionEmptyDaughter' completed for member 'jenkins-hbase4.apache.org,43091,1681268031804' in zk 2023-04-12 02:55:46,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:46,204 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionEmptyDaughter' received 'reached' from coordinator. 2023-04-12 02:55:46,204 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionEmptyDaughter' locally completed 2023-04-12 02:55:46,204 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionEmptyDaughter' completed for member 'jenkins-hbase4.apache.org,34579,1681268031719' in zk 2023-04-12 02:55:46,206 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionEmptyDaughter' has notified controller of completion 2023-04-12 02:55:46,206 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:55:46,206 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionEmptyDaughter' completed. 2023-04-12 02:55:46,206 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:46,206 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionEmptyDaughter' has notified controller of completion 2023-04-12 02:55:46,206 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:55:46,206 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionEmptyDaughter' completed. 2023-04-12 02:55:46,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:46,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 02:55:46,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 02:55:46,207 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionEmptyDaughter' has notified controller of completion 2023-04-12 02:55:46,207 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:55:46,207 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionEmptyDaughter' completed. 2023-04-12 02:55:46,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 02:55:46,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 02:55:46,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionEmptyDaughter 2023-04-12 02:55:46,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:46,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:46,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:46,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 02:55:46,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionEmptyDaughter 2023-04-12 02:55:46,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:46,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:46,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:46,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testSplitTableRegionEmptyDaughter' member 'jenkins-hbase4.apache.org,43091,1681268031804': 2023-04-12 02:55:46,210 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43091,1681268031804' released barrier for procedure'testSplitTableRegionEmptyDaughter', counting down latch. Waiting for 0 more 2023-04-12 02:55:46,210 INFO [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testSplitTableRegionEmptyDaughter' execution completed 2023-04-12 02:55:46,210 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 02:55:46,210 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 02:55:46,210 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testSplitTableRegionEmptyDaughter 2023-04-12 02:55:46,210 INFO [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testSplitTableRegionEmptyDaughterincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 02:55:46,212 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:46,212 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:55:46,212 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:46,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:46,212 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:46,212 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:55:46,212 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:46,212 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:46,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:46,212 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:46,212 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:55:46,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:46,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 02:55:46,212 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:46,212 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:55:46,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 02:55:46,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:55:46,213 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:55:46,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:55:46,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:46,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:46,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:46,213 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:55:46,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:55:46,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 02:55:46,213 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:46,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:46,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionEmptyDaughter 2023-04-12 02:55:46,215 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:46,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 02:55:46,215 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:46,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionEmptyDaughter 2023-04-12 02:55:46,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:46,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:46,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:46,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 02:55:46,217 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:46,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionEmptyDaughter 2023-04-12 02:55:46,217 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:46,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:46,217 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:46,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:46,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:46,220 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:55:46,220 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:55:46,220 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:55:46,220 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:55:46,220 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:55:46,220 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:55:46,220 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:46,220 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testSplitTableRegionEmptyDaughter' 2023-04-12 02:55:46,220 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 02:55:46,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:55:46,220 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:55:46,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:55:46,221 DEBUG [Listener at localhost/46667] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testSplitTableRegionEmptyDaughter'' to complete. (max 20000 ms per retry) 2023-04-12 02:55:46,220 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:55:46,220 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:46,221 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:55:46,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:55:46,221 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:55:46,221 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:55:46,221 DEBUG [Listener at localhost/46667] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 02:55:46,221 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:55:46,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:55:46,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:55:46,221 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:46,221 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:46,221 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:55:46,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:55:46,221 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:46,221 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:46,221 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:46,222 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:46,222 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:46,222 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:46,222 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:46,321 DEBUG [Listener at localhost/46667] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 02:55:46,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 02:55:46,334 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 02:55:46,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] procedure.ProcedureCoordinator(143): Procedure testSplitTableRegionEmptyDaughter was in running list but was completed. Accepting new attempt. 2023-04-12 02:55:46,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] procedure.ProcedureCoordinator(165): Submitting procedure testSplitTableRegionEmptyDaughter 2023-04-12 02:55:46,336 INFO [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testSplitTableRegionEmptyDaughter' 2023-04-12 02:55:46,336 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:55:46,336 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testSplitTableRegionEmptyDaughter' starting 'acquire' 2023-04-12 02:55:46,336 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testSplitTableRegionEmptyDaughter', kicking off acquire phase on members. 2023-04-12 02:55:46,336 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:46,336 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:46,338 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:55:46,338 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:55:46,338 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:55:46,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:55:46,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:55:46,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:55:46,338 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:46,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:55:46,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:55:46,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:55:46,339 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:46,339 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 02:55:46,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:46,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:46,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:46,339 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:46,339 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:46,339 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:46,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 02:55:46,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:46,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 02:55:46,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:46,340 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionEmptyDaughter 2023-04-12 02:55:46,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 02:55:46,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:46,340 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionEmptyDaughter 2023-04-12 02:55:46,340 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionEmptyDaughter 2023-04-12 02:55:46,340 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionEmptyDaughter is still present, removing 2023-04-12 02:55:46,340 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionEmptyDaughter 2023-04-12 02:55:46,340 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionEmptyDaughter is still present, removing 2023-04-12 02:55:46,340 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionEmptyDaughter' with timeout 60000ms 2023-04-12 02:55:46,341 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:55:46,341 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionEmptyDaughter' starting 'acquire' stage 2023-04-12 02:55:46,341 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionEmptyDaughter' locally acquired 2023-04-12 02:55:46,341 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34579,1681268031719' joining acquired barrier for procedure (testSplitTableRegionEmptyDaughter) in zk 2023-04-12 02:55:46,341 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionEmptyDaughter 2023-04-12 02:55:46,341 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionEmptyDaughter' with timeout 60000ms 2023-04-12 02:55:46,341 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:55:46,341 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionEmptyDaughter is still present, removing 2023-04-12 02:55:46,341 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionEmptyDaughter' starting 'acquire' stage 2023-04-12 02:55:46,341 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionEmptyDaughter' locally acquired 2023-04-12 02:55:46,341 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44679,1681268031772' joining acquired barrier for procedure (testSplitTableRegionEmptyDaughter) in zk 2023-04-12 02:55:46,341 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionEmptyDaughter 2023-04-12 02:55:46,341 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionEmptyDaughter' with timeout 60000ms 2023-04-12 02:55:46,341 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:55:46,341 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionEmptyDaughter' starting 'acquire' stage 2023-04-12 02:55:46,342 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 02:55:46,342 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 02:55:46,342 DEBUG [rs(jenkins-hbase4.apache.org,43091,1681268031804)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testSplitTableRegionEmptyDaughter,,1681268143910.7e32820b75c2980cc8d414d4b96996d4. 2023-04-12 02:55:46,342 DEBUG [rs(jenkins-hbase4.apache.org,43091,1681268031804)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testSplitTableRegionEmptyDaughter,,1681268143910.7e32820b75c2980cc8d414d4b96996d4. started... 2023-04-12 02:55:46,342 INFO [rs(jenkins-hbase4.apache.org,43091,1681268031804)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 7e32820b75c2980cc8d414d4b96996d4 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-04-12 02:55:46,343 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:46,344 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:46,344 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:46,344 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionEmptyDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 02:55:46,344 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:46,344 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionEmptyDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 02:55:46,356 INFO [rs(jenkins-hbase4.apache.org,43091,1681268031804)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=71 (bloomFilter=true), to=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/7e32820b75c2980cc8d414d4b96996d4/.tmp/cf1/51c0476f87c24b8ab6393791e8bea4dc 2023-04-12 02:55:46,369 INFO [rs(jenkins-hbase4.apache.org,43091,1681268031804)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=71 (bloomFilter=true), to=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/7e32820b75c2980cc8d414d4b96996d4/.tmp/cf2/9d166b5fa34e41e6bf53106735d2d210 2023-04-12 02:55:46,373 DEBUG [rs(jenkins-hbase4.apache.org,43091,1681268031804)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/7e32820b75c2980cc8d414d4b96996d4/.tmp/cf1/51c0476f87c24b8ab6393791e8bea4dc as hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/7e32820b75c2980cc8d414d4b96996d4/cf1/51c0476f87c24b8ab6393791e8bea4dc 2023-04-12 02:55:46,376 INFO [rs(jenkins-hbase4.apache.org,43091,1681268031804)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/7e32820b75c2980cc8d414d4b96996d4/cf1/51c0476f87c24b8ab6393791e8bea4dc, entries=10, sequenceid=71, filesize=5.0 K 2023-04-12 02:55:46,377 DEBUG [rs(jenkins-hbase4.apache.org,43091,1681268031804)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/7e32820b75c2980cc8d414d4b96996d4/.tmp/cf2/9d166b5fa34e41e6bf53106735d2d210 as hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/7e32820b75c2980cc8d414d4b96996d4/cf2/9d166b5fa34e41e6bf53106735d2d210 2023-04-12 02:55:46,380 INFO [rs(jenkins-hbase4.apache.org,43091,1681268031804)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/7e32820b75c2980cc8d414d4b96996d4/cf2/9d166b5fa34e41e6bf53106735d2d210, entries=10, sequenceid=71, filesize=5.0 K 2023-04-12 02:55:46,382 INFO [rs(jenkins-hbase4.apache.org,43091,1681268031804)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for 7e32820b75c2980cc8d414d4b96996d4 in 40ms, sequenceid=71, compaction requested=true 2023-04-12 02:55:46,383 DEBUG [rs(jenkins-hbase4.apache.org,43091,1681268031804)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 7e32820b75c2980cc8d414d4b96996d4: 2023-04-12 02:55:46,383 DEBUG [rs(jenkins-hbase4.apache.org,43091,1681268031804)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testSplitTableRegionEmptyDaughter,,1681268143910.7e32820b75c2980cc8d414d4b96996d4. 2023-04-12 02:55:46,383 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 02:55:46,383 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 02:55:46,383 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:46,383 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionEmptyDaughter' locally acquired 2023-04-12 02:55:46,383 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43091,1681268031804' joining acquired barrier for procedure (testSplitTableRegionEmptyDaughter) in zk 2023-04-12 02:55:46,384 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:46,384 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:46,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:46,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 02:55:46,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 02:55:46,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 02:55:46,386 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:46,386 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionEmptyDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 02:55:46,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 02:55:46,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionEmptyDaughter 2023-04-12 02:55:46,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:46,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:46,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:46,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 02:55:46,389 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,43091,1681268031804' joining acquired barrier for procedure 'testSplitTableRegionEmptyDaughter' on coordinator 2023-04-12 02:55:46,389 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6a572ad0[Count = 0] remaining members to acquire global barrier 2023-04-12 02:55:46,389 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testSplitTableRegionEmptyDaughter' starting 'in-barrier' execution. 2023-04-12 02:55:46,389 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:46,395 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:46,395 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:46,395 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:46,395 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:46,395 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:46,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:46,396 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:46,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:46,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:46,396 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:46,396 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 02:55:46,396 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionEmptyDaughter' received 'reached' from coordinator. 2023-04-12 02:55:46,396 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionEmptyDaughter' received 'reached' from coordinator. 2023-04-12 02:55:46,396 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionEmptyDaughter' locally completed 2023-04-12 02:55:46,396 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionEmptyDaughter' received 'reached' from coordinator. 2023-04-12 02:55:46,396 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionEmptyDaughter' locally completed 2023-04-12 02:55:46,396 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionEmptyDaughter' completed for member 'jenkins-hbase4.apache.org,44679,1681268031772' in zk 2023-04-12 02:55:46,396 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionEmptyDaughter' locally completed 2023-04-12 02:55:46,396 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionEmptyDaughter' completed for member 'jenkins-hbase4.apache.org,43091,1681268031804' in zk 2023-04-12 02:55:46,396 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionEmptyDaughter' completed for member 'jenkins-hbase4.apache.org,34579,1681268031719' in zk 2023-04-12 02:55:46,397 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionEmptyDaughter' has notified controller of completion 2023-04-12 02:55:46,397 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:55:46,398 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionEmptyDaughter' completed. 2023-04-12 02:55:46,399 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:46,399 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionEmptyDaughter' has notified controller of completion 2023-04-12 02:55:46,399 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionEmptyDaughter' has notified controller of completion 2023-04-12 02:55:46,399 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:55:46,399 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:55:46,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:46,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 02:55:46,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 02:55:46,399 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionEmptyDaughter' completed. 2023-04-12 02:55:46,399 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionEmptyDaughter' completed. 2023-04-12 02:55:46,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 02:55:46,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 02:55:46,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionEmptyDaughter 2023-04-12 02:55:46,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:46,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:46,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:46,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 02:55:46,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionEmptyDaughter 2023-04-12 02:55:46,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:46,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:46,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:46,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testSplitTableRegionEmptyDaughter' member 'jenkins-hbase4.apache.org,43091,1681268031804': 2023-04-12 02:55:46,405 INFO [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testSplitTableRegionEmptyDaughter' execution completed 2023-04-12 02:55:46,405 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 02:55:46,405 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 02:55:46,405 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,43091,1681268031804' released barrier for procedure'testSplitTableRegionEmptyDaughter', counting down latch. Waiting for 0 more 2023-04-12 02:55:46,405 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testSplitTableRegionEmptyDaughter 2023-04-12 02:55:46,405 INFO [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testSplitTableRegionEmptyDaughterincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 02:55:46,407 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:46,407 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:55:46,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:46,407 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:46,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:46,407 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:46,407 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:55:46,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:46,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:46,407 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:46,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:46,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:46,407 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:55:46,408 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:55:46,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:46,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 02:55:46,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:55:46,408 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:55:46,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 02:55:46,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:55:46,408 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:55:46,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:55:46,408 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:46,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:46,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:46,409 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:46,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:46,409 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:46,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 02:55:46,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionEmptyDaughter 2023-04-12 02:55:46,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 02:55:46,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionEmptyDaughter 2023-04-12 02:55:46,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:46,411 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:46,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:46,411 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:46,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:46,412 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:46,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 02:55:46,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionEmptyDaughter 2023-04-12 02:55:46,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:46,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:46,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:46,416 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:46,416 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:46,416 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:55:46,416 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:55:46,416 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:46,416 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:55:46,416 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:55:46,416 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:55:46,416 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testSplitTableRegionEmptyDaughter' 2023-04-12 02:55:46,416 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 02:55:46,416 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:55:46,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:55:46,416 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:55:46,416 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:55:46,416 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:55:46,417 DEBUG [Listener at localhost/46667] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testSplitTableRegionEmptyDaughter'' to complete. (max 20000 ms per retry) 2023-04-12 02:55:46,416 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:55:46,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:55:46,416 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:46,417 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:55:46,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:55:46,417 DEBUG [Listener at localhost/46667] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 02:55:46,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:55:46,417 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:46,417 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:55:46,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:55:46,417 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:46,418 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:55:46,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:55:46,418 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:46,418 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:46,418 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:46,418 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:46,418 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-12 02:55:46,517 DEBUG [Listener at localhost/46667] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 02:55:46,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 02:55:46,527 DEBUG [Listener at localhost/46667] assignment.SplitTableRegionProcedure(219): Splittable=true state=OPEN, location=jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:46,528 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(1029): Stored pid=128, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testSplitTableRegionEmptyDaughter, parent=7e32820b75c2980cc8d414d4b96996d4, daughterA=c0520a7ebf4c63c4a0116e7d97d4898b, daughterB=8dbe6ab5c0430ee2843a563720704953 2023-04-12 02:55:46,529 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=128, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testSplitTableRegionEmptyDaughter, parent=7e32820b75c2980cc8d414d4b96996d4, daughterA=c0520a7ebf4c63c4a0116e7d97d4898b, daughterB=8dbe6ab5c0430ee2843a563720704953 2023-04-12 02:55:46,530 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=128, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testSplitTableRegionEmptyDaughter, parent=7e32820b75c2980cc8d414d4b96996d4, daughterA=c0520a7ebf4c63c4a0116e7d97d4898b, daughterB=8dbe6ab5c0430ee2843a563720704953 2023-04-12 02:55:46,530 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=128, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testSplitTableRegionEmptyDaughter, parent=7e32820b75c2980cc8d414d4b96996d4, daughterA=c0520a7ebf4c63c4a0116e7d97d4898b, daughterB=8dbe6ab5c0430ee2843a563720704953 2023-04-12 02:55:46,532 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=129, ppid=128, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitTableRegionEmptyDaughter, region=7e32820b75c2980cc8d414d4b96996d4, UNASSIGN}] 2023-04-12 02:55:46,532 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=129, ppid=128, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitTableRegionEmptyDaughter, region=7e32820b75c2980cc8d414d4b96996d4, UNASSIGN 2023-04-12 02:55:46,539 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=129 updating hbase:meta row=7e32820b75c2980cc8d414d4b96996d4, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:46,539 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitTableRegionEmptyDaughter,,1681268143910.7e32820b75c2980cc8d414d4b96996d4.","families":{"info":[{"qualifier":"regioninfo","vlen":67,"tag":[],"timestamp":"1681268146539"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681268146539"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681268146539"}]},"ts":"1681268146539"} 2023-04-12 02:55:46,540 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=130, ppid=129, state=RUNNABLE; CloseRegionProcedure 7e32820b75c2980cc8d414d4b96996d4, server=jenkins-hbase4.apache.org,43091,1681268031804}] 2023-04-12 02:55:46,692 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 7e32820b75c2980cc8d414d4b96996d4 2023-04-12 02:55:46,692 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 7e32820b75c2980cc8d414d4b96996d4, disabling compactions & flushes 2023-04-12 02:55:46,692 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region testSplitTableRegionEmptyDaughter,,1681268143910.7e32820b75c2980cc8d414d4b96996d4. 2023-04-12 02:55:46,692 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testSplitTableRegionEmptyDaughter,,1681268143910.7e32820b75c2980cc8d414d4b96996d4. 2023-04-12 02:55:46,692 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on testSplitTableRegionEmptyDaughter,,1681268143910.7e32820b75c2980cc8d414d4b96996d4. after waiting 0 ms 2023-04-12 02:55:46,692 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region testSplitTableRegionEmptyDaughter,,1681268143910.7e32820b75c2980cc8d414d4b96996d4. 2023-04-12 02:55:46,693 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 7e32820b75c2980cc8d414d4b96996d4 2/2 column families, dataSize=480 B heapSize=2.13 KB 2023-04-12 02:55:46,708 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=240 B at sequenceid=82 (bloomFilter=true), to=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/7e32820b75c2980cc8d414d4b96996d4/.tmp/cf1/fddba85cfe21480c88a8f14c6018f209 2023-04-12 02:55:46,724 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=240 B at sequenceid=82 (bloomFilter=true), to=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/7e32820b75c2980cc8d414d4b96996d4/.tmp/cf2/0edadb780e4541c3b4438f10ddd7e003 2023-04-12 02:55:46,727 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/7e32820b75c2980cc8d414d4b96996d4/.tmp/cf1/fddba85cfe21480c88a8f14c6018f209 as hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/7e32820b75c2980cc8d414d4b96996d4/cf1/fddba85cfe21480c88a8f14c6018f209 2023-04-12 02:55:46,731 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/7e32820b75c2980cc8d414d4b96996d4/cf1/fddba85cfe21480c88a8f14c6018f209, entries=8, sequenceid=82, filesize=5.0 K 2023-04-12 02:55:46,731 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/7e32820b75c2980cc8d414d4b96996d4/.tmp/cf2/0edadb780e4541c3b4438f10ddd7e003 as hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/7e32820b75c2980cc8d414d4b96996d4/cf2/0edadb780e4541c3b4438f10ddd7e003 2023-04-12 02:55:46,735 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/7e32820b75c2980cc8d414d4b96996d4/cf2/0edadb780e4541c3b4438f10ddd7e003, entries=8, sequenceid=82, filesize=5.0 K 2023-04-12 02:55:46,736 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~480 B/480, heapSize ~2.09 KB/2144, currentSize=0 B/0 for 7e32820b75c2980cc8d414d4b96996d4 in 44ms, sequenceid=82, compaction requested=true 2023-04-12 02:55:46,740 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/7e32820b75c2980cc8d414d4b96996d4/recovered.edits/85.seqid, newMaxSeqId=85, maxSeqId=1 2023-04-12 02:55:46,741 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro 2023-04-12 02:55:46,742 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed testSplitTableRegionEmptyDaughter,,1681268143910.7e32820b75c2980cc8d414d4b96996d4. 2023-04-12 02:55:46,742 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 7e32820b75c2980cc8d414d4b96996d4: 2023-04-12 02:55:46,743 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 7e32820b75c2980cc8d414d4b96996d4 2023-04-12 02:55:46,743 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=129 updating hbase:meta row=7e32820b75c2980cc8d414d4b96996d4, regionState=CLOSED 2023-04-12 02:55:46,744 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testSplitTableRegionEmptyDaughter,,1681268143910.7e32820b75c2980cc8d414d4b96996d4.","families":{"info":[{"qualifier":"regioninfo","vlen":67,"tag":[],"timestamp":"1681268146743"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681268146743"}]},"ts":"1681268146743"} 2023-04-12 02:55:46,750 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=130, resume processing ppid=129 2023-04-12 02:55:46,750 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=130, ppid=129, state=SUCCESS; CloseRegionProcedure 7e32820b75c2980cc8d414d4b96996d4, server=jenkins-hbase4.apache.org,43091,1681268031804 in 205 msec 2023-04-12 02:55:46,752 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=129, resume processing ppid=128 2023-04-12 02:55:46,752 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=129, ppid=128, state=SUCCESS; TransitRegionStateProcedure table=testSplitTableRegionEmptyDaughter, region=7e32820b75c2980cc8d414d4b96996d4, UNASSIGN in 218 msec 2023-04-12 02:55:46,764 INFO [PEWorker-1] assignment.SplitTableRegionProcedure(694): pid=128 splitting 14 storefiles, region=7e32820b75c2980cc8d414d4b96996d4, threads=14 2023-04-12 02:55:46,765 DEBUG [StoreFileSplitter-pool-0] assignment.SplitTableRegionProcedure(776): pid=128 splitting started for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/7e32820b75c2980cc8d414d4b96996d4/cf2/0edadb780e4541c3b4438f10ddd7e003 for region: 7e32820b75c2980cc8d414d4b96996d4 2023-04-12 02:55:46,765 DEBUG [StoreFileSplitter-pool-1] assignment.SplitTableRegionProcedure(776): pid=128 splitting started for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/7e32820b75c2980cc8d414d4b96996d4/cf2/3f2612ae7a8b4c729ad90a32b95b9ce8 for region: 7e32820b75c2980cc8d414d4b96996d4 2023-04-12 02:55:46,765 DEBUG [StoreFileSplitter-pool-2] assignment.SplitTableRegionProcedure(776): pid=128 splitting started for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/7e32820b75c2980cc8d414d4b96996d4/cf2/5e06069d9e7543b390df83212345aa28 for region: 7e32820b75c2980cc8d414d4b96996d4 2023-04-12 02:55:46,765 DEBUG [StoreFileSplitter-pool-3] assignment.SplitTableRegionProcedure(776): pid=128 splitting started for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/7e32820b75c2980cc8d414d4b96996d4/cf2/8cf94287d6174b3daefe82fefab38588 for region: 7e32820b75c2980cc8d414d4b96996d4 2023-04-12 02:55:46,765 DEBUG [StoreFileSplitter-pool-4] assignment.SplitTableRegionProcedure(776): pid=128 splitting started for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/7e32820b75c2980cc8d414d4b96996d4/cf2/9d166b5fa34e41e6bf53106735d2d210 for region: 7e32820b75c2980cc8d414d4b96996d4 2023-04-12 02:55:46,766 DEBUG [StoreFileSplitter-pool-6] assignment.SplitTableRegionProcedure(776): pid=128 splitting started for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/7e32820b75c2980cc8d414d4b96996d4/cf2/ff655bed4b6d4c67a983029ee068984b for region: 7e32820b75c2980cc8d414d4b96996d4 2023-04-12 02:55:46,766 DEBUG [StoreFileSplitter-pool-7] assignment.SplitTableRegionProcedure(776): pid=128 splitting started for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/7e32820b75c2980cc8d414d4b96996d4/cf1/0039c871981747f0a784fb0e99655f62 for region: 7e32820b75c2980cc8d414d4b96996d4 2023-04-12 02:55:46,766 DEBUG [StoreFileSplitter-pool-8] assignment.SplitTableRegionProcedure(776): pid=128 splitting started for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/7e32820b75c2980cc8d414d4b96996d4/cf1/51c0476f87c24b8ab6393791e8bea4dc for region: 7e32820b75c2980cc8d414d4b96996d4 2023-04-12 02:55:46,766 DEBUG [StoreFileSplitter-pool-9] assignment.SplitTableRegionProcedure(776): pid=128 splitting started for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/7e32820b75c2980cc8d414d4b96996d4/cf1/abe57c09e0794d459254cbdc06a94880 for region: 7e32820b75c2980cc8d414d4b96996d4 2023-04-12 02:55:46,767 DEBUG [StoreFileSplitter-pool-5] assignment.SplitTableRegionProcedure(776): pid=128 splitting started for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/7e32820b75c2980cc8d414d4b96996d4/cf2/ed471e0ca1f449d7b18c5368407dcfa0 for region: 7e32820b75c2980cc8d414d4b96996d4 2023-04-12 02:55:46,767 DEBUG [StoreFileSplitter-pool-10] assignment.SplitTableRegionProcedure(776): pid=128 splitting started for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/7e32820b75c2980cc8d414d4b96996d4/cf1/db8da92f5ccc4286bc7911a7912d53db for region: 7e32820b75c2980cc8d414d4b96996d4 2023-04-12 02:55:46,767 DEBUG [StoreFileSplitter-pool-11] assignment.SplitTableRegionProcedure(776): pid=128 splitting started for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/7e32820b75c2980cc8d414d4b96996d4/cf1/e70af74642cc4f69ade829953a2cb4e3 for region: 7e32820b75c2980cc8d414d4b96996d4 2023-04-12 02:55:46,767 DEBUG [StoreFileSplitter-pool-12] assignment.SplitTableRegionProcedure(776): pid=128 splitting started for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/7e32820b75c2980cc8d414d4b96996d4/cf1/eb05d01415cc4933b6888f334b476175 for region: 7e32820b75c2980cc8d414d4b96996d4 2023-04-12 02:55:46,767 DEBUG [StoreFileSplitter-pool-13] assignment.SplitTableRegionProcedure(776): pid=128 splitting started for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/7e32820b75c2980cc8d414d4b96996d4/cf1/fddba85cfe21480c88a8f14c6018f209 for region: 7e32820b75c2980cc8d414d4b96996d4 2023-04-12 02:55:46,771 DEBUG [StoreFileSplitter-pool-8] regionserver.HRegionFileSystem(713): Will create HFileLink file for hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/7e32820b75c2980cc8d414d4b96996d4/cf1/51c0476f87c24b8ab6393791e8bea4dc, top=false 2023-04-12 02:55:46,771 DEBUG [StoreFileSplitter-pool-1] regionserver.HRegionFileSystem(713): Will create HFileLink file for hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/7e32820b75c2980cc8d414d4b96996d4/cf2/3f2612ae7a8b4c729ad90a32b95b9ce8, top=false 2023-04-12 02:55:46,772 DEBUG [StoreFileSplitter-pool-4] regionserver.HRegionFileSystem(713): Will create HFileLink file for hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/7e32820b75c2980cc8d414d4b96996d4/cf2/9d166b5fa34e41e6bf53106735d2d210, top=false 2023-04-12 02:55:46,773 DEBUG [StoreFileSplitter-pool-3] regionserver.HRegionFileSystem(713): Will create HFileLink file for hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/7e32820b75c2980cc8d414d4b96996d4/cf2/8cf94287d6174b3daefe82fefab38588, top=false 2023-04-12 02:55:46,775 DEBUG [StoreFileSplitter-pool-13] regionserver.HRegionFileSystem(713): Will create HFileLink file for hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/7e32820b75c2980cc8d414d4b96996d4/cf1/fddba85cfe21480c88a8f14c6018f209, top=false 2023-04-12 02:55:46,777 DEBUG [StoreFileSplitter-pool-2] regionserver.HRegionFileSystem(713): Will create HFileLink file for hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/7e32820b75c2980cc8d414d4b96996d4/cf2/5e06069d9e7543b390df83212345aa28, top=false 2023-04-12 02:55:46,778 DEBUG [StoreFileSplitter-pool-10] regionserver.HRegionFileSystem(713): Will create HFileLink file for hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/7e32820b75c2980cc8d414d4b96996d4/cf1/db8da92f5ccc4286bc7911a7912d53db, top=false 2023-04-12 02:55:46,783 DEBUG [StoreFileSplitter-pool-0] regionserver.HRegionFileSystem(713): Will create HFileLink file for hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/7e32820b75c2980cc8d414d4b96996d4/cf2/0edadb780e4541c3b4438f10ddd7e003, top=false 2023-04-12 02:55:46,784 DEBUG [StoreFileSplitter-pool-9] regionserver.HRegionFileSystem(713): Will create HFileLink file for hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/7e32820b75c2980cc8d414d4b96996d4/cf1/abe57c09e0794d459254cbdc06a94880, top=false 2023-04-12 02:55:46,784 DEBUG [StoreFileSplitter-pool-7] regionserver.HRegionFileSystem(713): Will create HFileLink file for hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/7e32820b75c2980cc8d414d4b96996d4/cf1/0039c871981747f0a784fb0e99655f62, top=false 2023-04-12 02:55:46,785 DEBUG [StoreFileSplitter-pool-5] regionserver.HRegionFileSystem(713): Will create HFileLink file for hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/7e32820b75c2980cc8d414d4b96996d4/cf2/ed471e0ca1f449d7b18c5368407dcfa0, top=false 2023-04-12 02:55:46,786 DEBUG [StoreFileSplitter-pool-11] regionserver.HRegionFileSystem(713): Will create HFileLink file for hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/7e32820b75c2980cc8d414d4b96996d4/cf1/e70af74642cc4f69ade829953a2cb4e3, top=false 2023-04-12 02:55:46,794 DEBUG [StoreFileSplitter-pool-6] regionserver.HRegionFileSystem(713): Will create HFileLink file for hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/7e32820b75c2980cc8d414d4b96996d4/cf2/ff655bed4b6d4c67a983029ee068984b, top=false 2023-04-12 02:55:46,806 DEBUG [StoreFileSplitter-pool-12] regionserver.HRegionFileSystem(713): Will create HFileLink file for hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/7e32820b75c2980cc8d414d4b96996d4/cf1/eb05d01415cc4933b6888f334b476175, top=false 2023-04-12 02:55:46,824 INFO [StoreFileSplitter-pool-3] regionserver.HRegionFileSystem(742): Created linkFile:hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/7e32820b75c2980cc8d414d4b96996d4/.splits/c0520a7ebf4c63c4a0116e7d97d4898b/cf2/testSplitTableRegionEmptyDaughter=7e32820b75c2980cc8d414d4b96996d4-8cf94287d6174b3daefe82fefab38588 for child: c0520a7ebf4c63c4a0116e7d97d4898b, parent: 7e32820b75c2980cc8d414d4b96996d4 2023-04-12 02:55:46,825 INFO [StoreFileSplitter-pool-10] regionserver.HRegionFileSystem(742): Created linkFile:hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/7e32820b75c2980cc8d414d4b96996d4/.splits/c0520a7ebf4c63c4a0116e7d97d4898b/cf1/testSplitTableRegionEmptyDaughter=7e32820b75c2980cc8d414d4b96996d4-db8da92f5ccc4286bc7911a7912d53db for child: c0520a7ebf4c63c4a0116e7d97d4898b, parent: 7e32820b75c2980cc8d414d4b96996d4 2023-04-12 02:55:46,825 INFO [StoreFileSplitter-pool-13] regionserver.HRegionFileSystem(742): Created linkFile:hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/7e32820b75c2980cc8d414d4b96996d4/.splits/c0520a7ebf4c63c4a0116e7d97d4898b/cf1/testSplitTableRegionEmptyDaughter=7e32820b75c2980cc8d414d4b96996d4-fddba85cfe21480c88a8f14c6018f209 for child: c0520a7ebf4c63c4a0116e7d97d4898b, parent: 7e32820b75c2980cc8d414d4b96996d4 2023-04-12 02:55:46,829 DEBUG [StoreFileSplitter-pool-10] assignment.SplitTableRegionProcedure(787): pid=128 splitting complete for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/7e32820b75c2980cc8d414d4b96996d4/cf1/db8da92f5ccc4286bc7911a7912d53db for region: 7e32820b75c2980cc8d414d4b96996d4 2023-04-12 02:55:46,829 DEBUG [StoreFileSplitter-pool-13] assignment.SplitTableRegionProcedure(787): pid=128 splitting complete for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/7e32820b75c2980cc8d414d4b96996d4/cf1/fddba85cfe21480c88a8f14c6018f209 for region: 7e32820b75c2980cc8d414d4b96996d4 2023-04-12 02:55:46,829 DEBUG [StoreFileSplitter-pool-3] assignment.SplitTableRegionProcedure(787): pid=128 splitting complete for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/7e32820b75c2980cc8d414d4b96996d4/cf2/8cf94287d6174b3daefe82fefab38588 for region: 7e32820b75c2980cc8d414d4b96996d4 2023-04-12 02:55:46,836 INFO [StoreFileSplitter-pool-5] regionserver.HRegionFileSystem(742): Created linkFile:hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/7e32820b75c2980cc8d414d4b96996d4/.splits/c0520a7ebf4c63c4a0116e7d97d4898b/cf2/testSplitTableRegionEmptyDaughter=7e32820b75c2980cc8d414d4b96996d4-ed471e0ca1f449d7b18c5368407dcfa0 for child: c0520a7ebf4c63c4a0116e7d97d4898b, parent: 7e32820b75c2980cc8d414d4b96996d4 2023-04-12 02:55:46,836 INFO [StoreFileSplitter-pool-11] regionserver.HRegionFileSystem(742): Created linkFile:hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/7e32820b75c2980cc8d414d4b96996d4/.splits/c0520a7ebf4c63c4a0116e7d97d4898b/cf1/testSplitTableRegionEmptyDaughter=7e32820b75c2980cc8d414d4b96996d4-e70af74642cc4f69ade829953a2cb4e3 for child: c0520a7ebf4c63c4a0116e7d97d4898b, parent: 7e32820b75c2980cc8d414d4b96996d4 2023-04-12 02:55:46,836 INFO [StoreFileSplitter-pool-2] regionserver.HRegionFileSystem(742): Created linkFile:hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/7e32820b75c2980cc8d414d4b96996d4/.splits/c0520a7ebf4c63c4a0116e7d97d4898b/cf2/testSplitTableRegionEmptyDaughter=7e32820b75c2980cc8d414d4b96996d4-5e06069d9e7543b390df83212345aa28 for child: c0520a7ebf4c63c4a0116e7d97d4898b, parent: 7e32820b75c2980cc8d414d4b96996d4 2023-04-12 02:55:46,846 DEBUG [StoreFileSplitter-pool-2] assignment.SplitTableRegionProcedure(787): pid=128 splitting complete for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/7e32820b75c2980cc8d414d4b96996d4/cf2/5e06069d9e7543b390df83212345aa28 for region: 7e32820b75c2980cc8d414d4b96996d4 2023-04-12 02:55:46,846 DEBUG [StoreFileSplitter-pool-5] assignment.SplitTableRegionProcedure(787): pid=128 splitting complete for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/7e32820b75c2980cc8d414d4b96996d4/cf2/ed471e0ca1f449d7b18c5368407dcfa0 for region: 7e32820b75c2980cc8d414d4b96996d4 2023-04-12 02:55:46,846 DEBUG [StoreFileSplitter-pool-11] assignment.SplitTableRegionProcedure(787): pid=128 splitting complete for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/7e32820b75c2980cc8d414d4b96996d4/cf1/e70af74642cc4f69ade829953a2cb4e3 for region: 7e32820b75c2980cc8d414d4b96996d4 2023-04-12 02:55:46,848 INFO [StoreFileSplitter-pool-9] regionserver.HRegionFileSystem(742): Created linkFile:hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/7e32820b75c2980cc8d414d4b96996d4/.splits/c0520a7ebf4c63c4a0116e7d97d4898b/cf1/testSplitTableRegionEmptyDaughter=7e32820b75c2980cc8d414d4b96996d4-abe57c09e0794d459254cbdc06a94880 for child: c0520a7ebf4c63c4a0116e7d97d4898b, parent: 7e32820b75c2980cc8d414d4b96996d4 2023-04-12 02:55:46,848 INFO [StoreFileSplitter-pool-12] regionserver.HRegionFileSystem(742): Created linkFile:hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/7e32820b75c2980cc8d414d4b96996d4/.splits/c0520a7ebf4c63c4a0116e7d97d4898b/cf1/testSplitTableRegionEmptyDaughter=7e32820b75c2980cc8d414d4b96996d4-eb05d01415cc4933b6888f334b476175 for child: c0520a7ebf4c63c4a0116e7d97d4898b, parent: 7e32820b75c2980cc8d414d4b96996d4 2023-04-12 02:55:46,848 INFO [StoreFileSplitter-pool-7] regionserver.HRegionFileSystem(742): Created linkFile:hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/7e32820b75c2980cc8d414d4b96996d4/.splits/c0520a7ebf4c63c4a0116e7d97d4898b/cf1/testSplitTableRegionEmptyDaughter=7e32820b75c2980cc8d414d4b96996d4-0039c871981747f0a784fb0e99655f62 for child: c0520a7ebf4c63c4a0116e7d97d4898b, parent: 7e32820b75c2980cc8d414d4b96996d4 2023-04-12 02:55:46,852 DEBUG [StoreFileSplitter-pool-7] assignment.SplitTableRegionProcedure(787): pid=128 splitting complete for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/7e32820b75c2980cc8d414d4b96996d4/cf1/0039c871981747f0a784fb0e99655f62 for region: 7e32820b75c2980cc8d414d4b96996d4 2023-04-12 02:55:46,852 DEBUG [StoreFileSplitter-pool-12] assignment.SplitTableRegionProcedure(787): pid=128 splitting complete for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/7e32820b75c2980cc8d414d4b96996d4/cf1/eb05d01415cc4933b6888f334b476175 for region: 7e32820b75c2980cc8d414d4b96996d4 2023-04-12 02:55:46,853 DEBUG [StoreFileSplitter-pool-9] assignment.SplitTableRegionProcedure(787): pid=128 splitting complete for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/7e32820b75c2980cc8d414d4b96996d4/cf1/abe57c09e0794d459254cbdc06a94880 for region: 7e32820b75c2980cc8d414d4b96996d4 2023-04-12 02:55:46,856 INFO [StoreFileSplitter-pool-1] regionserver.HRegionFileSystem(742): Created linkFile:hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/7e32820b75c2980cc8d414d4b96996d4/.splits/c0520a7ebf4c63c4a0116e7d97d4898b/cf2/testSplitTableRegionEmptyDaughter=7e32820b75c2980cc8d414d4b96996d4-3f2612ae7a8b4c729ad90a32b95b9ce8 for child: c0520a7ebf4c63c4a0116e7d97d4898b, parent: 7e32820b75c2980cc8d414d4b96996d4 2023-04-12 02:55:46,857 INFO [StoreFileSplitter-pool-8] regionserver.HRegionFileSystem(742): Created linkFile:hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/7e32820b75c2980cc8d414d4b96996d4/.splits/c0520a7ebf4c63c4a0116e7d97d4898b/cf1/testSplitTableRegionEmptyDaughter=7e32820b75c2980cc8d414d4b96996d4-51c0476f87c24b8ab6393791e8bea4dc for child: c0520a7ebf4c63c4a0116e7d97d4898b, parent: 7e32820b75c2980cc8d414d4b96996d4 2023-04-12 02:55:46,857 INFO [StoreFileSplitter-pool-6] regionserver.HRegionFileSystem(742): Created linkFile:hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/7e32820b75c2980cc8d414d4b96996d4/.splits/c0520a7ebf4c63c4a0116e7d97d4898b/cf2/testSplitTableRegionEmptyDaughter=7e32820b75c2980cc8d414d4b96996d4-ff655bed4b6d4c67a983029ee068984b for child: c0520a7ebf4c63c4a0116e7d97d4898b, parent: 7e32820b75c2980cc8d414d4b96996d4 2023-04-12 02:55:46,857 INFO [StoreFileSplitter-pool-0] regionserver.HRegionFileSystem(742): Created linkFile:hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/7e32820b75c2980cc8d414d4b96996d4/.splits/c0520a7ebf4c63c4a0116e7d97d4898b/cf2/testSplitTableRegionEmptyDaughter=7e32820b75c2980cc8d414d4b96996d4-0edadb780e4541c3b4438f10ddd7e003 for child: c0520a7ebf4c63c4a0116e7d97d4898b, parent: 7e32820b75c2980cc8d414d4b96996d4 2023-04-12 02:55:46,857 INFO [StoreFileSplitter-pool-4] regionserver.HRegionFileSystem(742): Created linkFile:hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/7e32820b75c2980cc8d414d4b96996d4/.splits/c0520a7ebf4c63c4a0116e7d97d4898b/cf2/testSplitTableRegionEmptyDaughter=7e32820b75c2980cc8d414d4b96996d4-9d166b5fa34e41e6bf53106735d2d210 for child: c0520a7ebf4c63c4a0116e7d97d4898b, parent: 7e32820b75c2980cc8d414d4b96996d4 2023-04-12 02:55:46,863 DEBUG [StoreFileSplitter-pool-1] assignment.SplitTableRegionProcedure(787): pid=128 splitting complete for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/7e32820b75c2980cc8d414d4b96996d4/cf2/3f2612ae7a8b4c729ad90a32b95b9ce8 for region: 7e32820b75c2980cc8d414d4b96996d4 2023-04-12 02:55:46,864 DEBUG [StoreFileSplitter-pool-8] assignment.SplitTableRegionProcedure(787): pid=128 splitting complete for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/7e32820b75c2980cc8d414d4b96996d4/cf1/51c0476f87c24b8ab6393791e8bea4dc for region: 7e32820b75c2980cc8d414d4b96996d4 2023-04-12 02:55:46,865 DEBUG [StoreFileSplitter-pool-4] assignment.SplitTableRegionProcedure(787): pid=128 splitting complete for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/7e32820b75c2980cc8d414d4b96996d4/cf2/9d166b5fa34e41e6bf53106735d2d210 for region: 7e32820b75c2980cc8d414d4b96996d4 2023-04-12 02:55:46,865 DEBUG [StoreFileSplitter-pool-6] assignment.SplitTableRegionProcedure(787): pid=128 splitting complete for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/7e32820b75c2980cc8d414d4b96996d4/cf2/ff655bed4b6d4c67a983029ee068984b for region: 7e32820b75c2980cc8d414d4b96996d4 2023-04-12 02:55:46,866 DEBUG [StoreFileSplitter-pool-0] assignment.SplitTableRegionProcedure(787): pid=128 splitting complete for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/7e32820b75c2980cc8d414d4b96996d4/cf2/0edadb780e4541c3b4438f10ddd7e003 for region: 7e32820b75c2980cc8d414d4b96996d4 2023-04-12 02:55:46,866 DEBUG [PEWorker-1] assignment.SplitTableRegionProcedure(755): pid=128 split storefiles for region 7e32820b75c2980cc8d414d4b96996d4 Daughter A: 14 storefiles, Daughter B: 0 storefiles. 2023-04-12 02:55:47,253 ERROR [BP-1247239137-172.31.14.131-1681268026808 heartbeating to localhost/127.0.0.1:44691] datanode.BPServiceActor(848): Exception in BPOfferService for Block pool BP-1247239137-172.31.14.131-1681268026808 (Datanode Uuid 79922b64-644d-4757-a58c-9707ee400c82) service to localhost/127.0.0.1:44691 java.lang.IllegalThreadStateException at java.lang.ThreadGroup.addUnstarted(ThreadGroup.java:867) at java.lang.Thread.init(Thread.java:407) at java.lang.Thread.init(Thread.java:351) at java.lang.Thread.(Thread.java:501) at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService$1.newThread(FsDatasetAsyncDiskService.java:102) at java.util.concurrent.ThreadPoolExecutor$Worker.(ThreadPoolExecutor.java:619) at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:932) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1367) at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService.execute(FsDatasetAsyncDiskService.java:172) at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService.deleteAsync(FsDatasetAsyncDiskService.java:221) at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.invalidate(FsDatasetImpl.java:2216) at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.invalidate(FsDatasetImpl.java:2140) at org.apache.hadoop.hdfs.server.datanode.BPOfferService.processCommandFromActive(BPOfferService.java:732) at org.apache.hadoop.hdfs.server.datanode.BPOfferService.processCommandFromActor(BPOfferService.java:679) at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.processCommand(BPServiceActor.java:880) at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:676) at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:846) at java.lang.Thread.run(Thread.java:750) 2023-04-12 02:55:47,318 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/c0520a7ebf4c63c4a0116e7d97d4898b/recovered.edits/85.seqid, newMaxSeqId=85, maxSeqId=-1 2023-04-12 02:55:47,321 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/8dbe6ab5c0430ee2843a563720704953/recovered.edits/85.seqid, newMaxSeqId=85, maxSeqId=-1 2023-04-12 02:55:47,323 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":3,"row":"testSplitTableRegionEmptyDaughter,,1681268143910.7e32820b75c2980cc8d414d4b96996d4.","families":{"info":[{"qualifier":"regioninfo","vlen":67,"tag":[],"timestamp":"1681268147323"},{"qualifier":"splitA","vlen":69,"tag":[],"timestamp":"1681268147323"},{"qualifier":"splitB","vlen":69,"tag":[],"timestamp":"1681268147323"}]},"ts":"1681268147323"} 2023-04-12 02:55:47,323 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":3,"row":"testSplitTableRegionEmptyDaughter,,1681268146527.c0520a7ebf4c63c4a0116e7d97d4898b.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1681268147323"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681268147323"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681268147323"}]},"ts":"1681268147323"} 2023-04-12 02:55:47,324 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":3,"row":"testSplitTableRegionEmptyDaughter,71,1681268146527.8dbe6ab5c0430ee2843a563720704953.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1681268147323"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681268147323"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681268147323"}]},"ts":"1681268147323"} 2023-04-12 02:55:47,329 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=131, ppid=128, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSplitTableRegionEmptyDaughter, region=c0520a7ebf4c63c4a0116e7d97d4898b, ASSIGN}, {pid=132, ppid=128, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSplitTableRegionEmptyDaughter, region=8dbe6ab5c0430ee2843a563720704953, ASSIGN}] 2023-04-12 02:55:47,330 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=132, ppid=128, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSplitTableRegionEmptyDaughter, region=8dbe6ab5c0430ee2843a563720704953, ASSIGN 2023-04-12 02:55:47,332 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=132, ppid=128, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testSplitTableRegionEmptyDaughter, region=8dbe6ab5c0430ee2843a563720704953, ASSIGN; state=SPLITTING_NEW, location=jenkins-hbase4.apache.org,43091,1681268031804; forceNewPlan=false, retain=false 2023-04-12 02:55:47,332 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=131, ppid=128, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSplitTableRegionEmptyDaughter, region=c0520a7ebf4c63c4a0116e7d97d4898b, ASSIGN 2023-04-12 02:55:47,333 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=131, ppid=128, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testSplitTableRegionEmptyDaughter, region=c0520a7ebf4c63c4a0116e7d97d4898b, ASSIGN; state=SPLITTING_NEW, location=jenkins-hbase4.apache.org,43091,1681268031804; forceNewPlan=false, retain=false 2023-04-12 02:55:47,482 INFO [jenkins-hbase4:40281] balancer.BaseLoadBalancer(1545): Reassigned 2 regions. 2 retained the pre-restart assignment. 2023-04-12 02:55:47,484 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=131 updating hbase:meta row=c0520a7ebf4c63c4a0116e7d97d4898b, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:47,484 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitTableRegionEmptyDaughter,,1681268146527.c0520a7ebf4c63c4a0116e7d97d4898b.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1681268147484"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681268147484"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681268147484"}]},"ts":"1681268147484"} 2023-04-12 02:55:47,485 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=133, ppid=131, state=RUNNABLE; OpenRegionProcedure c0520a7ebf4c63c4a0116e7d97d4898b, server=jenkins-hbase4.apache.org,43091,1681268031804}] 2023-04-12 02:55:47,490 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=132 updating hbase:meta row=8dbe6ab5c0430ee2843a563720704953, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:47,490 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitTableRegionEmptyDaughter,71,1681268146527.8dbe6ab5c0430ee2843a563720704953.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1681268147490"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681268147490"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681268147490"}]},"ts":"1681268147490"} 2023-04-12 02:55:47,496 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=134, ppid=132, state=RUNNABLE; OpenRegionProcedure 8dbe6ab5c0430ee2843a563720704953, server=jenkins-hbase4.apache.org,43091,1681268031804}] 2023-04-12 02:55:47,639 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testSplitTableRegionEmptyDaughter,,1681268146527.c0520a7ebf4c63c4a0116e7d97d4898b. 2023-04-12 02:55:47,640 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => c0520a7ebf4c63c4a0116e7d97d4898b, NAME => 'testSplitTableRegionEmptyDaughter,,1681268146527.c0520a7ebf4c63c4a0116e7d97d4898b.', STARTKEY => '', ENDKEY => '71'} 2023-04-12 02:55:47,640 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(174): System coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro loaded, priority=536870911. 2023-04-12 02:55:47,640 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testSplitTableRegionEmptyDaughter c0520a7ebf4c63c4a0116e7d97d4898b 2023-04-12 02:55:47,640 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated testSplitTableRegionEmptyDaughter,,1681268146527.c0520a7ebf4c63c4a0116e7d97d4898b.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-12 02:55:47,640 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for c0520a7ebf4c63c4a0116e7d97d4898b 2023-04-12 02:55:47,640 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for c0520a7ebf4c63c4a0116e7d97d4898b 2023-04-12 02:55:47,643 INFO [StoreOpener-c0520a7ebf4c63c4a0116e7d97d4898b-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf1 of region c0520a7ebf4c63c4a0116e7d97d4898b 2023-04-12 02:55:47,644 DEBUG [StoreOpener-c0520a7ebf4c63c4a0116e7d97d4898b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/c0520a7ebf4c63c4a0116e7d97d4898b/cf1 2023-04-12 02:55:47,644 DEBUG [StoreOpener-c0520a7ebf4c63c4a0116e7d97d4898b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/c0520a7ebf4c63c4a0116e7d97d4898b/cf1 2023-04-12 02:55:47,644 INFO [StoreOpener-c0520a7ebf4c63c4a0116e7d97d4898b-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region c0520a7ebf4c63c4a0116e7d97d4898b columnFamilyName cf1 2023-04-12 02:55:47,651 DEBUG [StoreOpener-c0520a7ebf4c63c4a0116e7d97d4898b-1] regionserver.HStore(539): loaded hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/c0520a7ebf4c63c4a0116e7d97d4898b/cf1/testSplitTableRegionEmptyDaughter=7e32820b75c2980cc8d414d4b96996d4-0039c871981747f0a784fb0e99655f62 2023-04-12 02:55:47,655 DEBUG [StoreOpener-c0520a7ebf4c63c4a0116e7d97d4898b-1] regionserver.HStore(539): loaded hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/c0520a7ebf4c63c4a0116e7d97d4898b/cf1/testSplitTableRegionEmptyDaughter=7e32820b75c2980cc8d414d4b96996d4-51c0476f87c24b8ab6393791e8bea4dc 2023-04-12 02:55:47,658 DEBUG [StoreOpener-c0520a7ebf4c63c4a0116e7d97d4898b-1] regionserver.HStore(539): loaded hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/c0520a7ebf4c63c4a0116e7d97d4898b/cf1/testSplitTableRegionEmptyDaughter=7e32820b75c2980cc8d414d4b96996d4-abe57c09e0794d459254cbdc06a94880 2023-04-12 02:55:47,660 DEBUG [StoreOpener-c0520a7ebf4c63c4a0116e7d97d4898b-1] regionserver.HStore(539): loaded hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/c0520a7ebf4c63c4a0116e7d97d4898b/cf1/testSplitTableRegionEmptyDaughter=7e32820b75c2980cc8d414d4b96996d4-db8da92f5ccc4286bc7911a7912d53db 2023-04-12 02:55:47,663 DEBUG [StoreOpener-c0520a7ebf4c63c4a0116e7d97d4898b-1] regionserver.HStore(539): loaded hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/c0520a7ebf4c63c4a0116e7d97d4898b/cf1/testSplitTableRegionEmptyDaughter=7e32820b75c2980cc8d414d4b96996d4-e70af74642cc4f69ade829953a2cb4e3 2023-04-12 02:55:47,666 DEBUG [StoreOpener-c0520a7ebf4c63c4a0116e7d97d4898b-1] regionserver.HStore(539): loaded hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/c0520a7ebf4c63c4a0116e7d97d4898b/cf1/testSplitTableRegionEmptyDaughter=7e32820b75c2980cc8d414d4b96996d4-eb05d01415cc4933b6888f334b476175 2023-04-12 02:55:47,669 DEBUG [StoreOpener-c0520a7ebf4c63c4a0116e7d97d4898b-1] regionserver.HStore(539): loaded hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/c0520a7ebf4c63c4a0116e7d97d4898b/cf1/testSplitTableRegionEmptyDaughter=7e32820b75c2980cc8d414d4b96996d4-fddba85cfe21480c88a8f14c6018f209 2023-04-12 02:55:47,669 INFO [StoreOpener-c0520a7ebf4c63c4a0116e7d97d4898b-1] regionserver.HStore(310): Store=c0520a7ebf4c63c4a0116e7d97d4898b/cf1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-12 02:55:47,669 INFO [StoreOpener-c0520a7ebf4c63c4a0116e7d97d4898b-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf2 of region c0520a7ebf4c63c4a0116e7d97d4898b 2023-04-12 02:55:47,670 DEBUG [StoreOpener-c0520a7ebf4c63c4a0116e7d97d4898b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/c0520a7ebf4c63c4a0116e7d97d4898b/cf2 2023-04-12 02:55:47,670 DEBUG [StoreOpener-c0520a7ebf4c63c4a0116e7d97d4898b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/c0520a7ebf4c63c4a0116e7d97d4898b/cf2 2023-04-12 02:55:47,670 INFO [StoreOpener-c0520a7ebf4c63c4a0116e7d97d4898b-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region c0520a7ebf4c63c4a0116e7d97d4898b columnFamilyName cf2 2023-04-12 02:55:47,678 DEBUG [StoreOpener-c0520a7ebf4c63c4a0116e7d97d4898b-1] regionserver.HStore(539): loaded hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/c0520a7ebf4c63c4a0116e7d97d4898b/cf2/testSplitTableRegionEmptyDaughter=7e32820b75c2980cc8d414d4b96996d4-0edadb780e4541c3b4438f10ddd7e003 2023-04-12 02:55:47,681 DEBUG [StoreOpener-c0520a7ebf4c63c4a0116e7d97d4898b-1] regionserver.HStore(539): loaded hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/c0520a7ebf4c63c4a0116e7d97d4898b/cf2/testSplitTableRegionEmptyDaughter=7e32820b75c2980cc8d414d4b96996d4-3f2612ae7a8b4c729ad90a32b95b9ce8 2023-04-12 02:55:47,684 DEBUG [StoreOpener-c0520a7ebf4c63c4a0116e7d97d4898b-1] regionserver.HStore(539): loaded hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/c0520a7ebf4c63c4a0116e7d97d4898b/cf2/testSplitTableRegionEmptyDaughter=7e32820b75c2980cc8d414d4b96996d4-5e06069d9e7543b390df83212345aa28 2023-04-12 02:55:47,688 DEBUG [StoreOpener-c0520a7ebf4c63c4a0116e7d97d4898b-1] regionserver.HStore(539): loaded hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/c0520a7ebf4c63c4a0116e7d97d4898b/cf2/testSplitTableRegionEmptyDaughter=7e32820b75c2980cc8d414d4b96996d4-8cf94287d6174b3daefe82fefab38588 2023-04-12 02:55:47,690 DEBUG [StoreOpener-c0520a7ebf4c63c4a0116e7d97d4898b-1] regionserver.HStore(539): loaded hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/c0520a7ebf4c63c4a0116e7d97d4898b/cf2/testSplitTableRegionEmptyDaughter=7e32820b75c2980cc8d414d4b96996d4-9d166b5fa34e41e6bf53106735d2d210 2023-04-12 02:55:47,693 DEBUG [StoreOpener-c0520a7ebf4c63c4a0116e7d97d4898b-1] regionserver.HStore(539): loaded hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/c0520a7ebf4c63c4a0116e7d97d4898b/cf2/testSplitTableRegionEmptyDaughter=7e32820b75c2980cc8d414d4b96996d4-ed471e0ca1f449d7b18c5368407dcfa0 2023-04-12 02:55:47,695 DEBUG [StoreOpener-c0520a7ebf4c63c4a0116e7d97d4898b-1] regionserver.HStore(539): loaded hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/c0520a7ebf4c63c4a0116e7d97d4898b/cf2/testSplitTableRegionEmptyDaughter=7e32820b75c2980cc8d414d4b96996d4-ff655bed4b6d4c67a983029ee068984b 2023-04-12 02:55:47,696 INFO [StoreOpener-c0520a7ebf4c63c4a0116e7d97d4898b-1] regionserver.HStore(310): Store=c0520a7ebf4c63c4a0116e7d97d4898b/cf2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-12 02:55:47,696 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/c0520a7ebf4c63c4a0116e7d97d4898b 2023-04-12 02:55:47,697 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/c0520a7ebf4c63c4a0116e7d97d4898b 2023-04-12 02:55:47,699 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testSplitTableRegionEmptyDaughter descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-04-12 02:55:47,700 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for c0520a7ebf4c63c4a0116e7d97d4898b 2023-04-12 02:55:47,700 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened c0520a7ebf4c63c4a0116e7d97d4898b; next sequenceid=86; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=67163067, jitterRate=8.076876401901245E-4}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-04-12 02:55:47,700 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for c0520a7ebf4c63c4a0116e7d97d4898b: 2023-04-12 02:55:47,701 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testSplitTableRegionEmptyDaughter,,1681268146527.c0520a7ebf4c63c4a0116e7d97d4898b., pid=133, masterSystemTime=1681268147637 2023-04-12 02:55:47,701 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: Opening Region; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 02:55:47,702 DEBUG [RS:2;jenkins-hbase4:43091-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-12 02:55:47,702 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: Opening Region; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 02:55:47,702 DEBUG [RS:2;jenkins-hbase4:43091-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-12 02:55:47,702 DEBUG [RS:2;jenkins-hbase4:43091-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 35728 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-04-12 02:55:47,703 DEBUG [RS:2;jenkins-hbase4:43091-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 35728 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-04-12 02:55:47,703 DEBUG [RS:2;jenkins-hbase4:43091-shortCompactions-0] regionserver.HStore(1912): c0520a7ebf4c63c4a0116e7d97d4898b/cf1 is initiating minor compaction (all files) 2023-04-12 02:55:47,703 DEBUG [RS:2;jenkins-hbase4:43091-longCompactions-0] regionserver.HStore(1912): c0520a7ebf4c63c4a0116e7d97d4898b/cf2 is initiating minor compaction (all files) 2023-04-12 02:55:47,703 INFO [RS:2;jenkins-hbase4:43091-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of c0520a7ebf4c63c4a0116e7d97d4898b/cf1 in testSplitTableRegionEmptyDaughter,,1681268146527.c0520a7ebf4c63c4a0116e7d97d4898b. 2023-04-12 02:55:47,703 INFO [RS:2;jenkins-hbase4:43091-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c0520a7ebf4c63c4a0116e7d97d4898b/cf2 in testSplitTableRegionEmptyDaughter,,1681268146527.c0520a7ebf4c63c4a0116e7d97d4898b. 2023-04-12 02:55:47,703 INFO [RS:2;jenkins-hbase4:43091-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/c0520a7ebf4c63c4a0116e7d97d4898b/cf1/testSplitTableRegionEmptyDaughter=7e32820b75c2980cc8d414d4b96996d4-abe57c09e0794d459254cbdc06a94880, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/c0520a7ebf4c63c4a0116e7d97d4898b/cf1/testSplitTableRegionEmptyDaughter=7e32820b75c2980cc8d414d4b96996d4-db8da92f5ccc4286bc7911a7912d53db, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/c0520a7ebf4c63c4a0116e7d97d4898b/cf1/testSplitTableRegionEmptyDaughter=7e32820b75c2980cc8d414d4b96996d4-eb05d01415cc4933b6888f334b476175, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/c0520a7ebf4c63c4a0116e7d97d4898b/cf1/testSplitTableRegionEmptyDaughter=7e32820b75c2980cc8d414d4b96996d4-e70af74642cc4f69ade829953a2cb4e3, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/c0520a7ebf4c63c4a0116e7d97d4898b/cf1/testSplitTableRegionEmptyDaughter=7e32820b75c2980cc8d414d4b96996d4-0039c871981747f0a784fb0e99655f62, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/c0520a7ebf4c63c4a0116e7d97d4898b/cf1/testSplitTableRegionEmptyDaughter=7e32820b75c2980cc8d414d4b96996d4-51c0476f87c24b8ab6393791e8bea4dc, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/c0520a7ebf4c63c4a0116e7d97d4898b/cf1/testSplitTableRegionEmptyDaughter=7e32820b75c2980cc8d414d4b96996d4-fddba85cfe21480c88a8f14c6018f209] into tmpdir=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/c0520a7ebf4c63c4a0116e7d97d4898b/.tmp, totalSize=34.9 K 2023-04-12 02:55:47,703 INFO [RS:2;jenkins-hbase4:43091-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/c0520a7ebf4c63c4a0116e7d97d4898b/cf2/testSplitTableRegionEmptyDaughter=7e32820b75c2980cc8d414d4b96996d4-8cf94287d6174b3daefe82fefab38588, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/c0520a7ebf4c63c4a0116e7d97d4898b/cf2/testSplitTableRegionEmptyDaughter=7e32820b75c2980cc8d414d4b96996d4-5e06069d9e7543b390df83212345aa28, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/c0520a7ebf4c63c4a0116e7d97d4898b/cf2/testSplitTableRegionEmptyDaughter=7e32820b75c2980cc8d414d4b96996d4-3f2612ae7a8b4c729ad90a32b95b9ce8, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/c0520a7ebf4c63c4a0116e7d97d4898b/cf2/testSplitTableRegionEmptyDaughter=7e32820b75c2980cc8d414d4b96996d4-ed471e0ca1f449d7b18c5368407dcfa0, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/c0520a7ebf4c63c4a0116e7d97d4898b/cf2/testSplitTableRegionEmptyDaughter=7e32820b75c2980cc8d414d4b96996d4-ff655bed4b6d4c67a983029ee068984b, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/c0520a7ebf4c63c4a0116e7d97d4898b/cf2/testSplitTableRegionEmptyDaughter=7e32820b75c2980cc8d414d4b96996d4-9d166b5fa34e41e6bf53106735d2d210, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/c0520a7ebf4c63c4a0116e7d97d4898b/cf2/testSplitTableRegionEmptyDaughter=7e32820b75c2980cc8d414d4b96996d4-0edadb780e4541c3b4438f10ddd7e003] into tmpdir=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/c0520a7ebf4c63c4a0116e7d97d4898b/.tmp, totalSize=34.9 K 2023-04-12 02:55:47,704 DEBUG [RS:2;jenkins-hbase4:43091-shortCompactions-0] compactions.Compactor(207): Compacting testSplitTableRegionEmptyDaughter=7e32820b75c2980cc8d414d4b96996d4-abe57c09e0794d459254cbdc06a94880, keycount=2, bloomtype=ROW, size=4.8 K, encoding=NONE, compression=NONE, seqNum=6, earliestPutTs=1681268144913 2023-04-12 02:55:47,704 DEBUG [RS:2;jenkins-hbase4:43091-longCompactions-0] compactions.Compactor(207): Compacting testSplitTableRegionEmptyDaughter=7e32820b75c2980cc8d414d4b96996d4-8cf94287d6174b3daefe82fefab38588, keycount=2, bloomtype=ROW, size=4.8 K, encoding=NONE, compression=NONE, seqNum=6, earliestPutTs=1681268144913 2023-04-12 02:55:47,704 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testSplitTableRegionEmptyDaughter,,1681268146527.c0520a7ebf4c63c4a0116e7d97d4898b. 2023-04-12 02:55:47,704 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testSplitTableRegionEmptyDaughter,,1681268146527.c0520a7ebf4c63c4a0116e7d97d4898b. 2023-04-12 02:55:47,704 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testSplitTableRegionEmptyDaughter,71,1681268146527.8dbe6ab5c0430ee2843a563720704953. 2023-04-12 02:55:47,704 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 8dbe6ab5c0430ee2843a563720704953, NAME => 'testSplitTableRegionEmptyDaughter,71,1681268146527.8dbe6ab5c0430ee2843a563720704953.', STARTKEY => '71', ENDKEY => ''} 2023-04-12 02:55:47,704 DEBUG [RS:2;jenkins-hbase4:43091-longCompactions-0] compactions.Compactor(207): Compacting testSplitTableRegionEmptyDaughter=7e32820b75c2980cc8d414d4b96996d4-5e06069d9e7543b390df83212345aa28, keycount=10, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=19, earliestPutTs=1681268145135 2023-04-12 02:55:47,705 DEBUG [RS:2;jenkins-hbase4:43091-shortCompactions-0] compactions.Compactor(207): Compacting testSplitTableRegionEmptyDaughter=7e32820b75c2980cc8d414d4b96996d4-db8da92f5ccc4286bc7911a7912d53db, keycount=10, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=19, earliestPutTs=1681268145135 2023-04-12 02:55:47,705 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(174): System coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro loaded, priority=536870911. 2023-04-12 02:55:47,705 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testSplitTableRegionEmptyDaughter 8dbe6ab5c0430ee2843a563720704953 2023-04-12 02:55:47,705 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated testSplitTableRegionEmptyDaughter,71,1681268146527.8dbe6ab5c0430ee2843a563720704953.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-12 02:55:47,705 DEBUG [RS:2;jenkins-hbase4:43091-longCompactions-0] compactions.Compactor(207): Compacting testSplitTableRegionEmptyDaughter=7e32820b75c2980cc8d414d4b96996d4-3f2612ae7a8b4c729ad90a32b95b9ce8, keycount=10, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=32, earliestPutTs=1681268145718 2023-04-12 02:55:47,705 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=131 updating hbase:meta row=c0520a7ebf4c63c4a0116e7d97d4898b, regionState=OPEN, openSeqNum=86, regionLocation=jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:47,705 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 8dbe6ab5c0430ee2843a563720704953 2023-04-12 02:55:47,705 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testSplitTableRegionEmptyDaughter,,1681268146527.c0520a7ebf4c63c4a0116e7d97d4898b.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1681268147705"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1681268147705"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681268147705"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681268147705"}]},"ts":"1681268147705"} 2023-04-12 02:55:47,705 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 8dbe6ab5c0430ee2843a563720704953 2023-04-12 02:55:47,706 DEBUG [RS:2;jenkins-hbase4:43091-shortCompactions-0] compactions.Compactor(207): Compacting testSplitTableRegionEmptyDaughter=7e32820b75c2980cc8d414d4b96996d4-eb05d01415cc4933b6888f334b476175, keycount=10, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=32, earliestPutTs=1681268145718 2023-04-12 02:55:47,706 DEBUG [RS:2;jenkins-hbase4:43091-longCompactions-0] compactions.Compactor(207): Compacting testSplitTableRegionEmptyDaughter=7e32820b75c2980cc8d414d4b96996d4-ed471e0ca1f449d7b18c5368407dcfa0, keycount=10, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=45, earliestPutTs=1681268145934 2023-04-12 02:55:47,706 DEBUG [RS:2;jenkins-hbase4:43091-longCompactions-0] compactions.Compactor(207): Compacting testSplitTableRegionEmptyDaughter=7e32820b75c2980cc8d414d4b96996d4-ff655bed4b6d4c67a983029ee068984b, keycount=10, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=58, earliestPutTs=1681268146137 2023-04-12 02:55:47,707 DEBUG [RS:2;jenkins-hbase4:43091-shortCompactions-0] compactions.Compactor(207): Compacting testSplitTableRegionEmptyDaughter=7e32820b75c2980cc8d414d4b96996d4-e70af74642cc4f69ade829953a2cb4e3, keycount=10, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=45, earliestPutTs=1681268145934 2023-04-12 02:55:47,707 DEBUG [RS:2;jenkins-hbase4:43091-longCompactions-0] compactions.Compactor(207): Compacting testSplitTableRegionEmptyDaughter=7e32820b75c2980cc8d414d4b96996d4-9d166b5fa34e41e6bf53106735d2d210, keycount=10, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=71, earliestPutTs=1681268146323 2023-04-12 02:55:47,707 INFO [StoreOpener-8dbe6ab5c0430ee2843a563720704953-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf1 of region 8dbe6ab5c0430ee2843a563720704953 2023-04-12 02:55:47,708 DEBUG [RS:2;jenkins-hbase4:43091-shortCompactions-0] compactions.Compactor(207): Compacting testSplitTableRegionEmptyDaughter=7e32820b75c2980cc8d414d4b96996d4-0039c871981747f0a784fb0e99655f62, keycount=10, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=58, earliestPutTs=1681268146137 2023-04-12 02:55:47,708 DEBUG [RS:2;jenkins-hbase4:43091-longCompactions-0] compactions.Compactor(207): Compacting testSplitTableRegionEmptyDaughter=7e32820b75c2980cc8d414d4b96996d4-0edadb780e4541c3b4438f10ddd7e003, keycount=8, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=82, earliestPutTs=1681268146519 2023-04-12 02:55:47,709 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=133, resume processing ppid=131 2023-04-12 02:55:47,709 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=133, ppid=131, state=SUCCESS; OpenRegionProcedure c0520a7ebf4c63c4a0116e7d97d4898b, server=jenkins-hbase4.apache.org,43091,1681268031804 in 222 msec 2023-04-12 02:55:47,709 DEBUG [RS:2;jenkins-hbase4:43091-shortCompactions-0] compactions.Compactor(207): Compacting testSplitTableRegionEmptyDaughter=7e32820b75c2980cc8d414d4b96996d4-51c0476f87c24b8ab6393791e8bea4dc, keycount=10, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=71, earliestPutTs=1681268146323 2023-04-12 02:55:47,710 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=131, ppid=128, state=SUCCESS; TransitRegionStateProcedure table=testSplitTableRegionEmptyDaughter, region=c0520a7ebf4c63c4a0116e7d97d4898b, ASSIGN in 380 msec 2023-04-12 02:55:47,710 DEBUG [RS:2;jenkins-hbase4:43091-shortCompactions-0] compactions.Compactor(207): Compacting testSplitTableRegionEmptyDaughter=7e32820b75c2980cc8d414d4b96996d4-fddba85cfe21480c88a8f14c6018f209, keycount=8, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=82, earliestPutTs=1681268146519 2023-04-12 02:55:47,712 DEBUG [StoreOpener-8dbe6ab5c0430ee2843a563720704953-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/8dbe6ab5c0430ee2843a563720704953/cf1 2023-04-12 02:55:47,712 DEBUG [StoreOpener-8dbe6ab5c0430ee2843a563720704953-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/8dbe6ab5c0430ee2843a563720704953/cf1 2023-04-12 02:55:47,713 INFO [StoreOpener-8dbe6ab5c0430ee2843a563720704953-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 8dbe6ab5c0430ee2843a563720704953 columnFamilyName cf1 2023-04-12 02:55:47,714 INFO [StoreOpener-8dbe6ab5c0430ee2843a563720704953-1] regionserver.HStore(310): Store=8dbe6ab5c0430ee2843a563720704953/cf1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-12 02:55:47,714 INFO [StoreOpener-8dbe6ab5c0430ee2843a563720704953-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf2 of region 8dbe6ab5c0430ee2843a563720704953 2023-04-12 02:55:47,718 DEBUG [StoreOpener-8dbe6ab5c0430ee2843a563720704953-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/8dbe6ab5c0430ee2843a563720704953/cf2 2023-04-12 02:55:47,718 DEBUG [StoreOpener-8dbe6ab5c0430ee2843a563720704953-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/8dbe6ab5c0430ee2843a563720704953/cf2 2023-04-12 02:55:47,720 INFO [StoreOpener-8dbe6ab5c0430ee2843a563720704953-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 8dbe6ab5c0430ee2843a563720704953 columnFamilyName cf2 2023-04-12 02:55:47,721 INFO [StoreOpener-8dbe6ab5c0430ee2843a563720704953-1] regionserver.HStore(310): Store=8dbe6ab5c0430ee2843a563720704953/cf2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-12 02:55:47,722 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/8dbe6ab5c0430ee2843a563720704953 2023-04-12 02:55:47,723 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/8dbe6ab5c0430ee2843a563720704953 2023-04-12 02:55:47,724 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testSplitTableRegionEmptyDaughter descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-04-12 02:55:47,725 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 8dbe6ab5c0430ee2843a563720704953 2023-04-12 02:55:47,726 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 8dbe6ab5c0430ee2843a563720704953; next sequenceid=86; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=61029695, jitterRate=-0.09058667719364166}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-04-12 02:55:47,726 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 8dbe6ab5c0430ee2843a563720704953: 2023-04-12 02:55:47,727 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testSplitTableRegionEmptyDaughter,71,1681268146527.8dbe6ab5c0430ee2843a563720704953., pid=134, masterSystemTime=1681268147637 2023-04-12 02:55:47,728 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testSplitTableRegionEmptyDaughter,71,1681268146527.8dbe6ab5c0430ee2843a563720704953. 2023-04-12 02:55:47,728 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testSplitTableRegionEmptyDaughter,71,1681268146527.8dbe6ab5c0430ee2843a563720704953. 2023-04-12 02:55:47,728 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=132 updating hbase:meta row=8dbe6ab5c0430ee2843a563720704953, regionState=OPEN, openSeqNum=86, regionLocation=jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:47,729 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testSplitTableRegionEmptyDaughter,71,1681268146527.8dbe6ab5c0430ee2843a563720704953.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1681268147728"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1681268147728"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681268147728"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681268147728"}]},"ts":"1681268147728"} 2023-04-12 02:55:47,731 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=134, resume processing ppid=132 2023-04-12 02:55:47,731 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=134, ppid=132, state=SUCCESS; OpenRegionProcedure 8dbe6ab5c0430ee2843a563720704953, server=jenkins-hbase4.apache.org,43091,1681268031804 in 234 msec 2023-04-12 02:55:47,733 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=132, resume processing ppid=128 2023-04-12 02:55:47,733 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=132, ppid=128, state=SUCCESS; TransitRegionStateProcedure table=testSplitTableRegionEmptyDaughter, region=8dbe6ab5c0430ee2843a563720704953, ASSIGN in 402 msec 2023-04-12 02:55:47,734 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=128, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionEmptyDaughter, parent=7e32820b75c2980cc8d414d4b96996d4, daughterA=c0520a7ebf4c63c4a0116e7d97d4898b, daughterB=8dbe6ab5c0430ee2843a563720704953 in 1.2060 sec 2023-04-12 02:55:47,747 INFO [RS:2;jenkins-hbase4:43091-longCompactions-0] throttle.PressureAwareThroughputController(145): c0520a7ebf4c63c4a0116e7d97d4898b#cf2#compaction#126 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 02:55:47,756 INFO [RS:2;jenkins-hbase4:43091-shortCompactions-0] throttle.PressureAwareThroughputController(145): c0520a7ebf4c63c4a0116e7d97d4898b#cf1#compaction#127 average throughput is 1.72 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 02:55:47,771 DEBUG [RS:2;jenkins-hbase4:43091-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/c0520a7ebf4c63c4a0116e7d97d4898b/.tmp/cf2/4210750adfba4468b3ef731d6811dbf1 as hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/c0520a7ebf4c63c4a0116e7d97d4898b/cf2/4210750adfba4468b3ef731d6811dbf1 2023-04-12 02:55:47,775 DEBUG [RS:2;jenkins-hbase4:43091-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/c0520a7ebf4c63c4a0116e7d97d4898b/.tmp/cf1/9fce144ef85146059b86bae95b74ee24 as hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/c0520a7ebf4c63c4a0116e7d97d4898b/cf1/9fce144ef85146059b86bae95b74ee24 2023-04-12 02:55:47,777 INFO [RS:2;jenkins-hbase4:43091-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in c0520a7ebf4c63c4a0116e7d97d4898b/cf2 of c0520a7ebf4c63c4a0116e7d97d4898b into 4210750adfba4468b3ef731d6811dbf1(size=7.3 K), total size for store is 7.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 02:55:47,777 DEBUG [RS:2;jenkins-hbase4:43091-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c0520a7ebf4c63c4a0116e7d97d4898b: 2023-04-12 02:55:47,777 INFO [RS:2;jenkins-hbase4:43091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=testSplitTableRegionEmptyDaughter,,1681268146527.c0520a7ebf4c63c4a0116e7d97d4898b., storeName=c0520a7ebf4c63c4a0116e7d97d4898b/cf2, priority=9, startTime=1681268147702; duration=0sec 2023-04-12 02:55:47,778 DEBUG [RS:2;jenkins-hbase4:43091-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 02:55:47,780 INFO [RS:2;jenkins-hbase4:43091-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in c0520a7ebf4c63c4a0116e7d97d4898b/cf1 of c0520a7ebf4c63c4a0116e7d97d4898b into 9fce144ef85146059b86bae95b74ee24(size=7.3 K), total size for store is 7.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 02:55:47,780 DEBUG [RS:2;jenkins-hbase4:43091-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for c0520a7ebf4c63c4a0116e7d97d4898b: 2023-04-12 02:55:47,780 INFO [RS:2;jenkins-hbase4:43091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=testSplitTableRegionEmptyDaughter,,1681268146527.c0520a7ebf4c63c4a0116e7d97d4898b., storeName=c0520a7ebf4c63c4a0116e7d97d4898b/cf1, priority=9, startTime=1681268147701; duration=0sec 2023-04-12 02:55:47,781 DEBUG [RS:2;jenkins-hbase4:43091-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 02:55:47,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43091] ipc.CallRunner(144): callId: 1003 service: ClientService methodName: Scan size: 157 connection: 172.31.14.131:38048 deadline: 1681268207781, exception=org.apache.hadoop.hbase.NotServingRegionException: testSplitTableRegionEmptyDaughter,,1681268143910.7e32820b75c2980cc8d414d4b96996d4. is not online on jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:47,893 DEBUG [Listener at localhost/46667] client.ClientScanner(240): Advancing internal scanner to startKey at '71', inclusive 2023-04-12 02:55:47,896 WARN [Listener at localhost/46667] procedure2.ProcedureTestingUtility(193): Set Kill before store update to: false 2023-04-12 02:55:47,898 INFO [Listener at localhost/46667] client.HBaseAdmin$15(890): Started disable of testSplitTableRegionEmptyDaughter 2023-04-12 02:55:47,899 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable testSplitTableRegionEmptyDaughter 2023-04-12 02:55:47,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] procedure2.ProcedureExecutor(1029): Stored pid=135, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testSplitTableRegionEmptyDaughter 2023-04-12 02:55:47,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] master.MasterRpcServices(1227): Checking to see if procedure is done pid=135 2023-04-12 02:55:47,902 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testSplitTableRegionEmptyDaughter","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681268147902"}]},"ts":"1681268147902"} 2023-04-12 02:55:47,903 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testSplitTableRegionEmptyDaughter, state=DISABLING in hbase:meta 2023-04-12 02:55:47,905 INFO [PEWorker-1] procedure.DisableTableProcedure(293): Set testSplitTableRegionEmptyDaughter to state=DISABLING 2023-04-12 02:55:47,905 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=136, ppid=135, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitTableRegionEmptyDaughter, region=c0520a7ebf4c63c4a0116e7d97d4898b, UNASSIGN}, {pid=137, ppid=135, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitTableRegionEmptyDaughter, region=8dbe6ab5c0430ee2843a563720704953, UNASSIGN}] 2023-04-12 02:55:47,906 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=137, ppid=135, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitTableRegionEmptyDaughter, region=8dbe6ab5c0430ee2843a563720704953, UNASSIGN 2023-04-12 02:55:47,906 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=137 updating hbase:meta row=8dbe6ab5c0430ee2843a563720704953, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:47,907 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitTableRegionEmptyDaughter,71,1681268146527.8dbe6ab5c0430ee2843a563720704953.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1681268147906"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681268147906"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681268147906"}]},"ts":"1681268147906"} 2023-04-12 02:55:47,908 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=138, ppid=137, state=RUNNABLE; CloseRegionProcedure 8dbe6ab5c0430ee2843a563720704953, server=jenkins-hbase4.apache.org,43091,1681268031804}] 2023-04-12 02:55:47,910 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=136, ppid=135, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitTableRegionEmptyDaughter, region=c0520a7ebf4c63c4a0116e7d97d4898b, UNASSIGN 2023-04-12 02:55:47,912 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=136 updating hbase:meta row=c0520a7ebf4c63c4a0116e7d97d4898b, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:47,912 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitTableRegionEmptyDaughter,,1681268146527.c0520a7ebf4c63c4a0116e7d97d4898b.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1681268147912"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681268147912"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681268147912"}]},"ts":"1681268147912"} 2023-04-12 02:55:47,914 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=139, ppid=136, state=RUNNABLE; CloseRegionProcedure c0520a7ebf4c63c4a0116e7d97d4898b, server=jenkins-hbase4.apache.org,43091,1681268031804}] 2023-04-12 02:55:48,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] master.MasterRpcServices(1227): Checking to see if procedure is done pid=135 2023-04-12 02:55:48,059 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 8dbe6ab5c0430ee2843a563720704953 2023-04-12 02:55:48,060 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 8dbe6ab5c0430ee2843a563720704953, disabling compactions & flushes 2023-04-12 02:55:48,061 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region testSplitTableRegionEmptyDaughter,71,1681268146527.8dbe6ab5c0430ee2843a563720704953. 2023-04-12 02:55:48,061 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testSplitTableRegionEmptyDaughter,71,1681268146527.8dbe6ab5c0430ee2843a563720704953. 2023-04-12 02:55:48,061 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on testSplitTableRegionEmptyDaughter,71,1681268146527.8dbe6ab5c0430ee2843a563720704953. after waiting 0 ms 2023-04-12 02:55:48,061 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region testSplitTableRegionEmptyDaughter,71,1681268146527.8dbe6ab5c0430ee2843a563720704953. 2023-04-12 02:55:48,064 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/8dbe6ab5c0430ee2843a563720704953/recovered.edits/88.seqid, newMaxSeqId=88, maxSeqId=85 2023-04-12 02:55:48,065 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro 2023-04-12 02:55:48,065 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed testSplitTableRegionEmptyDaughter,71,1681268146527.8dbe6ab5c0430ee2843a563720704953. 2023-04-12 02:55:48,065 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 8dbe6ab5c0430ee2843a563720704953: 2023-04-12 02:55:48,066 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 8dbe6ab5c0430ee2843a563720704953 2023-04-12 02:55:48,066 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close c0520a7ebf4c63c4a0116e7d97d4898b 2023-04-12 02:55:48,067 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing c0520a7ebf4c63c4a0116e7d97d4898b, disabling compactions & flushes 2023-04-12 02:55:48,067 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region testSplitTableRegionEmptyDaughter,,1681268146527.c0520a7ebf4c63c4a0116e7d97d4898b. 2023-04-12 02:55:48,067 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testSplitTableRegionEmptyDaughter,,1681268146527.c0520a7ebf4c63c4a0116e7d97d4898b. 2023-04-12 02:55:48,067 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on testSplitTableRegionEmptyDaughter,,1681268146527.c0520a7ebf4c63c4a0116e7d97d4898b. after waiting 0 ms 2023-04-12 02:55:48,067 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region testSplitTableRegionEmptyDaughter,,1681268146527.c0520a7ebf4c63c4a0116e7d97d4898b. 2023-04-12 02:55:48,067 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=137 updating hbase:meta row=8dbe6ab5c0430ee2843a563720704953, regionState=CLOSED 2023-04-12 02:55:48,067 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testSplitTableRegionEmptyDaughter,71,1681268146527.8dbe6ab5c0430ee2843a563720704953.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1681268148067"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681268148067"}]},"ts":"1681268148067"} 2023-04-12 02:55:48,067 DEBUG [StoreCloser-testSplitTableRegionEmptyDaughter,,1681268146527.c0520a7ebf4c63c4a0116e7d97d4898b.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/c0520a7ebf4c63c4a0116e7d97d4898b/cf1/testSplitTableRegionEmptyDaughter=7e32820b75c2980cc8d414d4b96996d4-abe57c09e0794d459254cbdc06a94880, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/c0520a7ebf4c63c4a0116e7d97d4898b/cf1/testSplitTableRegionEmptyDaughter=7e32820b75c2980cc8d414d4b96996d4-db8da92f5ccc4286bc7911a7912d53db, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/c0520a7ebf4c63c4a0116e7d97d4898b/cf1/testSplitTableRegionEmptyDaughter=7e32820b75c2980cc8d414d4b96996d4-eb05d01415cc4933b6888f334b476175, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/c0520a7ebf4c63c4a0116e7d97d4898b/cf1/testSplitTableRegionEmptyDaughter=7e32820b75c2980cc8d414d4b96996d4-e70af74642cc4f69ade829953a2cb4e3, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/c0520a7ebf4c63c4a0116e7d97d4898b/cf1/testSplitTableRegionEmptyDaughter=7e32820b75c2980cc8d414d4b96996d4-0039c871981747f0a784fb0e99655f62, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/c0520a7ebf4c63c4a0116e7d97d4898b/cf1/testSplitTableRegionEmptyDaughter=7e32820b75c2980cc8d414d4b96996d4-51c0476f87c24b8ab6393791e8bea4dc, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/c0520a7ebf4c63c4a0116e7d97d4898b/cf1/testSplitTableRegionEmptyDaughter=7e32820b75c2980cc8d414d4b96996d4-fddba85cfe21480c88a8f14c6018f209] to archive 2023-04-12 02:55:48,068 DEBUG [StoreCloser-testSplitTableRegionEmptyDaughter,,1681268146527.c0520a7ebf4c63c4a0116e7d97d4898b.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-12 02:55:48,070 DEBUG [StoreCloser-testSplitTableRegionEmptyDaughter,,1681268146527.c0520a7ebf4c63c4a0116e7d97d4898b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/c0520a7ebf4c63c4a0116e7d97d4898b/cf1/testSplitTableRegionEmptyDaughter=7e32820b75c2980cc8d414d4b96996d4-abe57c09e0794d459254cbdc06a94880 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegionEmptyDaughter/c0520a7ebf4c63c4a0116e7d97d4898b/cf1/testSplitTableRegionEmptyDaughter=7e32820b75c2980cc8d414d4b96996d4-abe57c09e0794d459254cbdc06a94880 2023-04-12 02:55:48,070 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=138, resume processing ppid=137 2023-04-12 02:55:48,071 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=138, ppid=137, state=SUCCESS; CloseRegionProcedure 8dbe6ab5c0430ee2843a563720704953, server=jenkins-hbase4.apache.org,43091,1681268031804 in 161 msec 2023-04-12 02:55:48,072 DEBUG [StoreCloser-testSplitTableRegionEmptyDaughter,,1681268146527.c0520a7ebf4c63c4a0116e7d97d4898b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/c0520a7ebf4c63c4a0116e7d97d4898b/cf1/testSplitTableRegionEmptyDaughter=7e32820b75c2980cc8d414d4b96996d4-db8da92f5ccc4286bc7911a7912d53db to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegionEmptyDaughter/c0520a7ebf4c63c4a0116e7d97d4898b/cf1/testSplitTableRegionEmptyDaughter=7e32820b75c2980cc8d414d4b96996d4-db8da92f5ccc4286bc7911a7912d53db 2023-04-12 02:55:48,072 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=137, ppid=135, state=SUCCESS; TransitRegionStateProcedure table=testSplitTableRegionEmptyDaughter, region=8dbe6ab5c0430ee2843a563720704953, UNASSIGN in 166 msec 2023-04-12 02:55:48,074 DEBUG [StoreCloser-testSplitTableRegionEmptyDaughter,,1681268146527.c0520a7ebf4c63c4a0116e7d97d4898b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/c0520a7ebf4c63c4a0116e7d97d4898b/cf1/testSplitTableRegionEmptyDaughter=7e32820b75c2980cc8d414d4b96996d4-eb05d01415cc4933b6888f334b476175 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegionEmptyDaughter/c0520a7ebf4c63c4a0116e7d97d4898b/cf1/testSplitTableRegionEmptyDaughter=7e32820b75c2980cc8d414d4b96996d4-eb05d01415cc4933b6888f334b476175 2023-04-12 02:55:48,076 DEBUG [StoreCloser-testSplitTableRegionEmptyDaughter,,1681268146527.c0520a7ebf4c63c4a0116e7d97d4898b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/c0520a7ebf4c63c4a0116e7d97d4898b/cf1/testSplitTableRegionEmptyDaughter=7e32820b75c2980cc8d414d4b96996d4-e70af74642cc4f69ade829953a2cb4e3 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegionEmptyDaughter/c0520a7ebf4c63c4a0116e7d97d4898b/cf1/testSplitTableRegionEmptyDaughter=7e32820b75c2980cc8d414d4b96996d4-e70af74642cc4f69ade829953a2cb4e3 2023-04-12 02:55:48,078 DEBUG [StoreCloser-testSplitTableRegionEmptyDaughter,,1681268146527.c0520a7ebf4c63c4a0116e7d97d4898b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/c0520a7ebf4c63c4a0116e7d97d4898b/cf1/testSplitTableRegionEmptyDaughter=7e32820b75c2980cc8d414d4b96996d4-0039c871981747f0a784fb0e99655f62 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegionEmptyDaughter/c0520a7ebf4c63c4a0116e7d97d4898b/cf1/testSplitTableRegionEmptyDaughter=7e32820b75c2980cc8d414d4b96996d4-0039c871981747f0a784fb0e99655f62 2023-04-12 02:55:48,080 DEBUG [StoreCloser-testSplitTableRegionEmptyDaughter,,1681268146527.c0520a7ebf4c63c4a0116e7d97d4898b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/c0520a7ebf4c63c4a0116e7d97d4898b/cf1/testSplitTableRegionEmptyDaughter=7e32820b75c2980cc8d414d4b96996d4-51c0476f87c24b8ab6393791e8bea4dc to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegionEmptyDaughter/c0520a7ebf4c63c4a0116e7d97d4898b/cf1/testSplitTableRegionEmptyDaughter=7e32820b75c2980cc8d414d4b96996d4-51c0476f87c24b8ab6393791e8bea4dc 2023-04-12 02:55:48,084 DEBUG [StoreCloser-testSplitTableRegionEmptyDaughter,,1681268146527.c0520a7ebf4c63c4a0116e7d97d4898b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/c0520a7ebf4c63c4a0116e7d97d4898b/cf1/testSplitTableRegionEmptyDaughter=7e32820b75c2980cc8d414d4b96996d4-fddba85cfe21480c88a8f14c6018f209 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegionEmptyDaughter/c0520a7ebf4c63c4a0116e7d97d4898b/cf1/testSplitTableRegionEmptyDaughter=7e32820b75c2980cc8d414d4b96996d4-fddba85cfe21480c88a8f14c6018f209 2023-04-12 02:55:48,087 DEBUG [StoreCloser-testSplitTableRegionEmptyDaughter,,1681268146527.c0520a7ebf4c63c4a0116e7d97d4898b.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/c0520a7ebf4c63c4a0116e7d97d4898b/cf2/testSplitTableRegionEmptyDaughter=7e32820b75c2980cc8d414d4b96996d4-8cf94287d6174b3daefe82fefab38588, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/c0520a7ebf4c63c4a0116e7d97d4898b/cf2/testSplitTableRegionEmptyDaughter=7e32820b75c2980cc8d414d4b96996d4-5e06069d9e7543b390df83212345aa28, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/c0520a7ebf4c63c4a0116e7d97d4898b/cf2/testSplitTableRegionEmptyDaughter=7e32820b75c2980cc8d414d4b96996d4-3f2612ae7a8b4c729ad90a32b95b9ce8, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/c0520a7ebf4c63c4a0116e7d97d4898b/cf2/testSplitTableRegionEmptyDaughter=7e32820b75c2980cc8d414d4b96996d4-ed471e0ca1f449d7b18c5368407dcfa0, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/c0520a7ebf4c63c4a0116e7d97d4898b/cf2/testSplitTableRegionEmptyDaughter=7e32820b75c2980cc8d414d4b96996d4-ff655bed4b6d4c67a983029ee068984b, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/c0520a7ebf4c63c4a0116e7d97d4898b/cf2/testSplitTableRegionEmptyDaughter=7e32820b75c2980cc8d414d4b96996d4-9d166b5fa34e41e6bf53106735d2d210, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/c0520a7ebf4c63c4a0116e7d97d4898b/cf2/testSplitTableRegionEmptyDaughter=7e32820b75c2980cc8d414d4b96996d4-0edadb780e4541c3b4438f10ddd7e003] to archive 2023-04-12 02:55:48,087 DEBUG [StoreCloser-testSplitTableRegionEmptyDaughter,,1681268146527.c0520a7ebf4c63c4a0116e7d97d4898b.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-12 02:55:48,090 DEBUG [StoreCloser-testSplitTableRegionEmptyDaughter,,1681268146527.c0520a7ebf4c63c4a0116e7d97d4898b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/c0520a7ebf4c63c4a0116e7d97d4898b/cf2/testSplitTableRegionEmptyDaughter=7e32820b75c2980cc8d414d4b96996d4-8cf94287d6174b3daefe82fefab38588 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegionEmptyDaughter/c0520a7ebf4c63c4a0116e7d97d4898b/cf2/testSplitTableRegionEmptyDaughter=7e32820b75c2980cc8d414d4b96996d4-8cf94287d6174b3daefe82fefab38588 2023-04-12 02:55:48,091 DEBUG [StoreCloser-testSplitTableRegionEmptyDaughter,,1681268146527.c0520a7ebf4c63c4a0116e7d97d4898b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/c0520a7ebf4c63c4a0116e7d97d4898b/cf2/testSplitTableRegionEmptyDaughter=7e32820b75c2980cc8d414d4b96996d4-5e06069d9e7543b390df83212345aa28 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegionEmptyDaughter/c0520a7ebf4c63c4a0116e7d97d4898b/cf2/testSplitTableRegionEmptyDaughter=7e32820b75c2980cc8d414d4b96996d4-5e06069d9e7543b390df83212345aa28 2023-04-12 02:55:48,093 DEBUG [StoreCloser-testSplitTableRegionEmptyDaughter,,1681268146527.c0520a7ebf4c63c4a0116e7d97d4898b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/c0520a7ebf4c63c4a0116e7d97d4898b/cf2/testSplitTableRegionEmptyDaughter=7e32820b75c2980cc8d414d4b96996d4-3f2612ae7a8b4c729ad90a32b95b9ce8 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegionEmptyDaughter/c0520a7ebf4c63c4a0116e7d97d4898b/cf2/testSplitTableRegionEmptyDaughter=7e32820b75c2980cc8d414d4b96996d4-3f2612ae7a8b4c729ad90a32b95b9ce8 2023-04-12 02:55:48,094 DEBUG [StoreCloser-testSplitTableRegionEmptyDaughter,,1681268146527.c0520a7ebf4c63c4a0116e7d97d4898b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/c0520a7ebf4c63c4a0116e7d97d4898b/cf2/testSplitTableRegionEmptyDaughter=7e32820b75c2980cc8d414d4b96996d4-ed471e0ca1f449d7b18c5368407dcfa0 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegionEmptyDaughter/c0520a7ebf4c63c4a0116e7d97d4898b/cf2/testSplitTableRegionEmptyDaughter=7e32820b75c2980cc8d414d4b96996d4-ed471e0ca1f449d7b18c5368407dcfa0 2023-04-12 02:55:48,095 DEBUG [StoreCloser-testSplitTableRegionEmptyDaughter,,1681268146527.c0520a7ebf4c63c4a0116e7d97d4898b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/c0520a7ebf4c63c4a0116e7d97d4898b/cf2/testSplitTableRegionEmptyDaughter=7e32820b75c2980cc8d414d4b96996d4-ff655bed4b6d4c67a983029ee068984b to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegionEmptyDaughter/c0520a7ebf4c63c4a0116e7d97d4898b/cf2/testSplitTableRegionEmptyDaughter=7e32820b75c2980cc8d414d4b96996d4-ff655bed4b6d4c67a983029ee068984b 2023-04-12 02:55:48,097 DEBUG [StoreCloser-testSplitTableRegionEmptyDaughter,,1681268146527.c0520a7ebf4c63c4a0116e7d97d4898b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/c0520a7ebf4c63c4a0116e7d97d4898b/cf2/testSplitTableRegionEmptyDaughter=7e32820b75c2980cc8d414d4b96996d4-9d166b5fa34e41e6bf53106735d2d210 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegionEmptyDaughter/c0520a7ebf4c63c4a0116e7d97d4898b/cf2/testSplitTableRegionEmptyDaughter=7e32820b75c2980cc8d414d4b96996d4-9d166b5fa34e41e6bf53106735d2d210 2023-04-12 02:55:48,098 DEBUG [StoreCloser-testSplitTableRegionEmptyDaughter,,1681268146527.c0520a7ebf4c63c4a0116e7d97d4898b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/c0520a7ebf4c63c4a0116e7d97d4898b/cf2/testSplitTableRegionEmptyDaughter=7e32820b75c2980cc8d414d4b96996d4-0edadb780e4541c3b4438f10ddd7e003 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegionEmptyDaughter/c0520a7ebf4c63c4a0116e7d97d4898b/cf2/testSplitTableRegionEmptyDaughter=7e32820b75c2980cc8d414d4b96996d4-0edadb780e4541c3b4438f10ddd7e003 2023-04-12 02:55:48,107 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegionEmptyDaughter/c0520a7ebf4c63c4a0116e7d97d4898b/recovered.edits/90.seqid, newMaxSeqId=90, maxSeqId=85 2023-04-12 02:55:48,108 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro 2023-04-12 02:55:48,108 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed testSplitTableRegionEmptyDaughter,,1681268146527.c0520a7ebf4c63c4a0116e7d97d4898b. 2023-04-12 02:55:48,108 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for c0520a7ebf4c63c4a0116e7d97d4898b: 2023-04-12 02:55:48,110 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed c0520a7ebf4c63c4a0116e7d97d4898b 2023-04-12 02:55:48,110 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=136 updating hbase:meta row=c0520a7ebf4c63c4a0116e7d97d4898b, regionState=CLOSED 2023-04-12 02:55:48,110 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testSplitTableRegionEmptyDaughter,,1681268146527.c0520a7ebf4c63c4a0116e7d97d4898b.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1681268148110"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681268148110"}]},"ts":"1681268148110"} 2023-04-12 02:55:48,113 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=139, resume processing ppid=136 2023-04-12 02:55:48,114 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=139, ppid=136, state=SUCCESS; CloseRegionProcedure c0520a7ebf4c63c4a0116e7d97d4898b, server=jenkins-hbase4.apache.org,43091,1681268031804 in 199 msec 2023-04-12 02:55:48,115 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=136, resume processing ppid=135 2023-04-12 02:55:48,115 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=136, ppid=135, state=SUCCESS; TransitRegionStateProcedure table=testSplitTableRegionEmptyDaughter, region=c0520a7ebf4c63c4a0116e7d97d4898b, UNASSIGN in 209 msec 2023-04-12 02:55:48,117 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testSplitTableRegionEmptyDaughter","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681268148117"}]},"ts":"1681268148117"} 2023-04-12 02:55:48,118 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testSplitTableRegionEmptyDaughter, state=DISABLED in hbase:meta 2023-04-12 02:55:48,125 INFO [PEWorker-1] procedure.DisableTableProcedure(305): Set testSplitTableRegionEmptyDaughter to state=DISABLED 2023-04-12 02:55:48,127 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=135, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionEmptyDaughter in 227 msec 2023-04-12 02:55:48,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] master.MasterRpcServices(1227): Checking to see if procedure is done pid=135 2023-04-12 02:55:48,204 INFO [Listener at localhost/46667] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testSplitTableRegionEmptyDaughter, procId: 135 completed 2023-04-12 02:55:48,205 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete testSplitTableRegionEmptyDaughter 2023-04-12 02:55:48,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] procedure2.ProcedureExecutor(1029): Stored pid=140, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=testSplitTableRegionEmptyDaughter 2023-04-12 02:55:48,208 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=140, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=testSplitTableRegionEmptyDaughter 2023-04-12 02:55:48,209 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=140, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=testSplitTableRegionEmptyDaughter 2023-04-12 02:55:48,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] master.MasterRpcServices(1227): Checking to see if procedure is done pid=140 2023-04-12 02:55:48,213 DEBUG [HFileArchiver-9] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitTableRegionEmptyDaughter/7e32820b75c2980cc8d414d4b96996d4 2023-04-12 02:55:48,213 DEBUG [HFileArchiver-10] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitTableRegionEmptyDaughter/c0520a7ebf4c63c4a0116e7d97d4898b 2023-04-12 02:55:48,213 DEBUG [HFileArchiver-11] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitTableRegionEmptyDaughter/8dbe6ab5c0430ee2843a563720704953 2023-04-12 02:55:48,215 DEBUG [HFileArchiver-11] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitTableRegionEmptyDaughter/8dbe6ab5c0430ee2843a563720704953/cf1, FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitTableRegionEmptyDaughter/8dbe6ab5c0430ee2843a563720704953/cf2, FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitTableRegionEmptyDaughter/8dbe6ab5c0430ee2843a563720704953/recovered.edits] 2023-04-12 02:55:48,215 DEBUG [HFileArchiver-10] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitTableRegionEmptyDaughter/c0520a7ebf4c63c4a0116e7d97d4898b/cf1, FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitTableRegionEmptyDaughter/c0520a7ebf4c63c4a0116e7d97d4898b/cf2, FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitTableRegionEmptyDaughter/c0520a7ebf4c63c4a0116e7d97d4898b/recovered.edits] 2023-04-12 02:55:48,215 DEBUG [HFileArchiver-9] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitTableRegionEmptyDaughter/7e32820b75c2980cc8d414d4b96996d4/cf1, FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitTableRegionEmptyDaughter/7e32820b75c2980cc8d414d4b96996d4/cf2, FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitTableRegionEmptyDaughter/7e32820b75c2980cc8d414d4b96996d4/recovered.edits] 2023-04-12 02:55:48,218 DEBUG [HFileArchiver-10] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitTableRegionEmptyDaughter/c0520a7ebf4c63c4a0116e7d97d4898b/cf1/9fce144ef85146059b86bae95b74ee24 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegionEmptyDaughter/c0520a7ebf4c63c4a0116e7d97d4898b/cf1/9fce144ef85146059b86bae95b74ee24 2023-04-12 02:55:48,219 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitTableRegionEmptyDaughter/7e32820b75c2980cc8d414d4b96996d4/cf1/0039c871981747f0a784fb0e99655f62 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegionEmptyDaughter/7e32820b75c2980cc8d414d4b96996d4/cf1/0039c871981747f0a784fb0e99655f62 2023-04-12 02:55:48,222 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitTableRegionEmptyDaughter/7e32820b75c2980cc8d414d4b96996d4/cf1/51c0476f87c24b8ab6393791e8bea4dc to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegionEmptyDaughter/7e32820b75c2980cc8d414d4b96996d4/cf1/51c0476f87c24b8ab6393791e8bea4dc 2023-04-12 02:55:48,222 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitTableRegionEmptyDaughter/8dbe6ab5c0430ee2843a563720704953/recovered.edits/88.seqid to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegionEmptyDaughter/8dbe6ab5c0430ee2843a563720704953/recovered.edits/88.seqid 2023-04-12 02:55:48,223 DEBUG [HFileArchiver-10] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitTableRegionEmptyDaughter/c0520a7ebf4c63c4a0116e7d97d4898b/cf2/4210750adfba4468b3ef731d6811dbf1 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegionEmptyDaughter/c0520a7ebf4c63c4a0116e7d97d4898b/cf2/4210750adfba4468b3ef731d6811dbf1 2023-04-12 02:55:48,223 DEBUG [HFileArchiver-11] backup.HFileArchiver(596): Deleted hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitTableRegionEmptyDaughter/8dbe6ab5c0430ee2843a563720704953 2023-04-12 02:55:48,224 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitTableRegionEmptyDaughter/7e32820b75c2980cc8d414d4b96996d4/cf1/abe57c09e0794d459254cbdc06a94880 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegionEmptyDaughter/7e32820b75c2980cc8d414d4b96996d4/cf1/abe57c09e0794d459254cbdc06a94880 2023-04-12 02:55:48,225 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitTableRegionEmptyDaughter/7e32820b75c2980cc8d414d4b96996d4/cf1/db8da92f5ccc4286bc7911a7912d53db to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegionEmptyDaughter/7e32820b75c2980cc8d414d4b96996d4/cf1/db8da92f5ccc4286bc7911a7912d53db 2023-04-12 02:55:48,226 DEBUG [HFileArchiver-10] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitTableRegionEmptyDaughter/c0520a7ebf4c63c4a0116e7d97d4898b/recovered.edits/90.seqid to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegionEmptyDaughter/c0520a7ebf4c63c4a0116e7d97d4898b/recovered.edits/90.seqid 2023-04-12 02:55:48,227 DEBUG [HFileArchiver-10] backup.HFileArchiver(596): Deleted hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitTableRegionEmptyDaughter/c0520a7ebf4c63c4a0116e7d97d4898b 2023-04-12 02:55:48,228 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitTableRegionEmptyDaughter/7e32820b75c2980cc8d414d4b96996d4/cf1/e70af74642cc4f69ade829953a2cb4e3 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegionEmptyDaughter/7e32820b75c2980cc8d414d4b96996d4/cf1/e70af74642cc4f69ade829953a2cb4e3 2023-04-12 02:55:48,229 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitTableRegionEmptyDaughter/7e32820b75c2980cc8d414d4b96996d4/cf1/eb05d01415cc4933b6888f334b476175 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegionEmptyDaughter/7e32820b75c2980cc8d414d4b96996d4/cf1/eb05d01415cc4933b6888f334b476175 2023-04-12 02:55:48,231 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitTableRegionEmptyDaughter/7e32820b75c2980cc8d414d4b96996d4/cf1/fddba85cfe21480c88a8f14c6018f209 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegionEmptyDaughter/7e32820b75c2980cc8d414d4b96996d4/cf1/fddba85cfe21480c88a8f14c6018f209 2023-04-12 02:55:48,233 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitTableRegionEmptyDaughter/7e32820b75c2980cc8d414d4b96996d4/cf2/0edadb780e4541c3b4438f10ddd7e003 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegionEmptyDaughter/7e32820b75c2980cc8d414d4b96996d4/cf2/0edadb780e4541c3b4438f10ddd7e003 2023-04-12 02:55:48,234 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitTableRegionEmptyDaughter/7e32820b75c2980cc8d414d4b96996d4/cf2/3f2612ae7a8b4c729ad90a32b95b9ce8 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegionEmptyDaughter/7e32820b75c2980cc8d414d4b96996d4/cf2/3f2612ae7a8b4c729ad90a32b95b9ce8 2023-04-12 02:55:48,235 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitTableRegionEmptyDaughter/7e32820b75c2980cc8d414d4b96996d4/cf2/5e06069d9e7543b390df83212345aa28 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegionEmptyDaughter/7e32820b75c2980cc8d414d4b96996d4/cf2/5e06069d9e7543b390df83212345aa28 2023-04-12 02:55:48,237 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitTableRegionEmptyDaughter/7e32820b75c2980cc8d414d4b96996d4/cf2/8cf94287d6174b3daefe82fefab38588 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegionEmptyDaughter/7e32820b75c2980cc8d414d4b96996d4/cf2/8cf94287d6174b3daefe82fefab38588 2023-04-12 02:55:48,238 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitTableRegionEmptyDaughter/7e32820b75c2980cc8d414d4b96996d4/cf2/9d166b5fa34e41e6bf53106735d2d210 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegionEmptyDaughter/7e32820b75c2980cc8d414d4b96996d4/cf2/9d166b5fa34e41e6bf53106735d2d210 2023-04-12 02:55:48,240 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitTableRegionEmptyDaughter/7e32820b75c2980cc8d414d4b96996d4/cf2/ed471e0ca1f449d7b18c5368407dcfa0 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegionEmptyDaughter/7e32820b75c2980cc8d414d4b96996d4/cf2/ed471e0ca1f449d7b18c5368407dcfa0 2023-04-12 02:55:48,241 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitTableRegionEmptyDaughter/7e32820b75c2980cc8d414d4b96996d4/cf2/ff655bed4b6d4c67a983029ee068984b to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegionEmptyDaughter/7e32820b75c2980cc8d414d4b96996d4/cf2/ff655bed4b6d4c67a983029ee068984b 2023-04-12 02:55:48,243 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitTableRegionEmptyDaughter/7e32820b75c2980cc8d414d4b96996d4/recovered.edits/85.seqid to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegionEmptyDaughter/7e32820b75c2980cc8d414d4b96996d4/recovered.edits/85.seqid 2023-04-12 02:55:48,244 DEBUG [HFileArchiver-9] backup.HFileArchiver(596): Deleted hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitTableRegionEmptyDaughter/7e32820b75c2980cc8d414d4b96996d4 2023-04-12 02:55:48,244 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived testSplitTableRegionEmptyDaughter regions 2023-04-12 02:55:48,246 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=140, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=testSplitTableRegionEmptyDaughter 2023-04-12 02:55:48,248 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 3 rows of testSplitTableRegionEmptyDaughter from hbase:meta 2023-04-12 02:55:48,250 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'testSplitTableRegionEmptyDaughter' descriptor. 2023-04-12 02:55:48,251 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=140, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=testSplitTableRegionEmptyDaughter 2023-04-12 02:55:48,251 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'testSplitTableRegionEmptyDaughter' from region states. 2023-04-12 02:55:48,251 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testSplitTableRegionEmptyDaughter,,1681268143910.7e32820b75c2980cc8d414d4b96996d4.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1681268148251"}]},"ts":"9223372036854775807"} 2023-04-12 02:55:48,251 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testSplitTableRegionEmptyDaughter,,1681268146527.c0520a7ebf4c63c4a0116e7d97d4898b.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1681268148251"}]},"ts":"9223372036854775807"} 2023-04-12 02:55:48,251 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testSplitTableRegionEmptyDaughter,71,1681268146527.8dbe6ab5c0430ee2843a563720704953.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1681268148251"}]},"ts":"9223372036854775807"} 2023-04-12 02:55:48,252 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 3 regions from META 2023-04-12 02:55:48,252 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 7e32820b75c2980cc8d414d4b96996d4, NAME => 'testSplitTableRegionEmptyDaughter,,1681268143910.7e32820b75c2980cc8d414d4b96996d4.', STARTKEY => '', ENDKEY => ''}, {ENCODED => c0520a7ebf4c63c4a0116e7d97d4898b, NAME => 'testSplitTableRegionEmptyDaughter,,1681268146527.c0520a7ebf4c63c4a0116e7d97d4898b.', STARTKEY => '', ENDKEY => '71'}, {ENCODED => 8dbe6ab5c0430ee2843a563720704953, NAME => 'testSplitTableRegionEmptyDaughter,71,1681268146527.8dbe6ab5c0430ee2843a563720704953.', STARTKEY => '71', ENDKEY => ''}] 2023-04-12 02:55:48,253 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'testSplitTableRegionEmptyDaughter' as deleted. 2023-04-12 02:55:48,253 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testSplitTableRegionEmptyDaughter","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1681268148253"}]},"ts":"9223372036854775807"} 2023-04-12 02:55:48,254 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table testSplitTableRegionEmptyDaughter state from META 2023-04-12 02:55:48,260 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=140, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=testSplitTableRegionEmptyDaughter 2023-04-12 02:55:48,261 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=140, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionEmptyDaughter in 55 msec 2023-04-12 02:55:48,302 ERROR [BP-1247239137-172.31.14.131-1681268026808 heartbeating to localhost/127.0.0.1:44691] datanode.BPServiceActor(848): Exception in BPOfferService for Block pool BP-1247239137-172.31.14.131-1681268026808 (Datanode Uuid 569a15e2-91b7-43b5-bb2b-d70e35911578) service to localhost/127.0.0.1:44691 java.lang.IllegalThreadStateException at java.lang.ThreadGroup.addUnstarted(ThreadGroup.java:867) at java.lang.Thread.init(Thread.java:407) at java.lang.Thread.init(Thread.java:351) at java.lang.Thread.(Thread.java:501) at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService$1.newThread(FsDatasetAsyncDiskService.java:102) at java.util.concurrent.ThreadPoolExecutor$Worker.(ThreadPoolExecutor.java:619) at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:932) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1367) at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService.execute(FsDatasetAsyncDiskService.java:172) at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService.deleteAsync(FsDatasetAsyncDiskService.java:221) at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.invalidate(FsDatasetImpl.java:2216) at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.invalidate(FsDatasetImpl.java:2140) at org.apache.hadoop.hdfs.server.datanode.BPOfferService.processCommandFromActive(BPOfferService.java:732) at org.apache.hadoop.hdfs.server.datanode.BPOfferService.processCommandFromActor(BPOfferService.java:679) at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.processCommand(BPServiceActor.java:880) at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:676) at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:846) at java.lang.Thread.run(Thread.java:750) 2023-04-12 02:55:48,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] master.MasterRpcServices(1227): Checking to see if procedure is done pid=140 2023-04-12 02:55:48,310 INFO [Listener at localhost/46667] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:testSplitTableRegionEmptyDaughter, procId: 140 completed 2023-04-12 02:55:48,327 INFO [Listener at localhost/46667] hbase.ResourceChecker(175): after: master.assignment.TestSplitTableRegionProcedure#testSplitTableRegionEmptyDaughter Thread=373 (was 375), OpenFileDescriptor=819 (was 821), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=362 (was 350) - SystemLoadAverage LEAK? -, ProcessCount=170 (was 171), AvailableMemoryMB=767 (was 817) 2023-04-12 02:55:48,347 INFO [Listener at localhost/46667] hbase.ResourceChecker(147): before: master.assignment.TestSplitTableRegionProcedure#testSplitTableRegion Thread=373, OpenFileDescriptor=819, MaxFileDescriptor=60000, SystemLoadAverage=362, ProcessCount=170, AvailableMemoryMB=767 2023-04-12 02:55:48,347 WARN [Listener at localhost/46667] procedure2.ProcedureTestingUtility(193): Set Kill before store update to: false 2023-04-12 02:55:48,350 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/balancer 2023-04-12 02:55:48,350 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] master.MasterRpcServices(492): Client=jenkins//172.31.14.131 set balanceSwitch=false 2023-04-12 02:55:48,351 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(1029): Stored pid=141, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=testSplitTableRegion 2023-04-12 02:55:48,353 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=141, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=testSplitTableRegion execute state=CREATE_TABLE_PRE_OPERATION 2023-04-12 02:55:48,354 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=141, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=testSplitTableRegion execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-12 02:55:48,357 DEBUG [HFileArchiver-12] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4 2023-04-12 02:55:48,358 DEBUG [HFileArchiver-12] backup.HFileArchiver(153): Directory hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4 empty. 2023-04-12 02:55:48,358 DEBUG [HFileArchiver-12] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4 2023-04-12 02:55:48,358 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived testSplitTableRegion regions 2023-04-12 02:55:48,381 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitTableRegion/.tabledesc/.tableinfo.0000000001 2023-04-12 02:55:48,384 INFO [RegionOpenAndInit-testSplitTableRegion-pool-0] regionserver.HRegion(7675): creating {ENCODED => c426152f6c5cdf7d09ab747442f08ea4, NAME => 'testSplitTableRegion,,1681268148351.c426152f6c5cdf7d09ab747442f08ea4.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='testSplitTableRegion', {NAME => 'cf1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'cf2', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp 2023-04-12 02:55:48,403 DEBUG [RegionOpenAndInit-testSplitTableRegion-pool-0] regionserver.HRegion(866): Instantiated testSplitTableRegion,,1681268148351.c426152f6c5cdf7d09ab747442f08ea4.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-12 02:55:48,403 DEBUG [RegionOpenAndInit-testSplitTableRegion-pool-0] regionserver.HRegion(1604): Closing c426152f6c5cdf7d09ab747442f08ea4, disabling compactions & flushes 2023-04-12 02:55:48,403 INFO [RegionOpenAndInit-testSplitTableRegion-pool-0] regionserver.HRegion(1626): Closing region testSplitTableRegion,,1681268148351.c426152f6c5cdf7d09ab747442f08ea4. 2023-04-12 02:55:48,403 DEBUG [RegionOpenAndInit-testSplitTableRegion-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testSplitTableRegion,,1681268148351.c426152f6c5cdf7d09ab747442f08ea4. 2023-04-12 02:55:48,403 DEBUG [RegionOpenAndInit-testSplitTableRegion-pool-0] regionserver.HRegion(1714): Acquired close lock on testSplitTableRegion,,1681268148351.c426152f6c5cdf7d09ab747442f08ea4. after waiting 0 ms 2023-04-12 02:55:48,403 DEBUG [RegionOpenAndInit-testSplitTableRegion-pool-0] regionserver.HRegion(1724): Updates disabled for region testSplitTableRegion,,1681268148351.c426152f6c5cdf7d09ab747442f08ea4. 2023-04-12 02:55:48,403 INFO [RegionOpenAndInit-testSplitTableRegion-pool-0] regionserver.HRegion(1838): Closed testSplitTableRegion,,1681268148351.c426152f6c5cdf7d09ab747442f08ea4. 2023-04-12 02:55:48,403 DEBUG [RegionOpenAndInit-testSplitTableRegion-pool-0] regionserver.HRegion(1558): Region close journal for c426152f6c5cdf7d09ab747442f08ea4: 2023-04-12 02:55:48,406 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=141, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=testSplitTableRegion execute state=CREATE_TABLE_ADD_TO_META 2023-04-12 02:55:48,407 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testSplitTableRegion,,1681268148351.c426152f6c5cdf7d09ab747442f08ea4.","families":{"info":[{"qualifier":"regioninfo","vlen":54,"tag":[],"timestamp":"1681268148406"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681268148406"}]},"ts":"1681268148406"} 2023-04-12 02:55:48,408 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-12 02:55:48,409 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=141, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=testSplitTableRegion execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-12 02:55:48,409 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testSplitTableRegion","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681268148409"}]},"ts":"1681268148409"} 2023-04-12 02:55:48,411 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testSplitTableRegion, state=ENABLING in hbase:meta 2023-04-12 02:55:48,415 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-04-12 02:55:48,416 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-04-12 02:55:48,416 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-04-12 02:55:48,416 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-04-12 02:55:48,416 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-04-12 02:55:48,416 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=142, ppid=141, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSplitTableRegion, region=c426152f6c5cdf7d09ab747442f08ea4, ASSIGN}] 2023-04-12 02:55:48,418 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=142, ppid=141, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSplitTableRegion, region=c426152f6c5cdf7d09ab747442f08ea4, ASSIGN 2023-04-12 02:55:48,418 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=142, ppid=141, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testSplitTableRegion, region=c426152f6c5cdf7d09ab747442f08ea4, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,34579,1681268031719; forceNewPlan=false, retain=false 2023-04-12 02:55:48,569 INFO [jenkins-hbase4:40281] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-04-12 02:55:48,570 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=142 updating hbase:meta row=c426152f6c5cdf7d09ab747442f08ea4, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:48,570 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitTableRegion,,1681268148351.c426152f6c5cdf7d09ab747442f08ea4.","families":{"info":[{"qualifier":"regioninfo","vlen":54,"tag":[],"timestamp":"1681268148570"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681268148570"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681268148570"}]},"ts":"1681268148570"} 2023-04-12 02:55:48,571 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=143, ppid=142, state=RUNNABLE; OpenRegionProcedure c426152f6c5cdf7d09ab747442f08ea4, server=jenkins-hbase4.apache.org,34579,1681268031719}] 2023-04-12 02:55:48,727 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testSplitTableRegion,,1681268148351.c426152f6c5cdf7d09ab747442f08ea4. 2023-04-12 02:55:48,727 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => c426152f6c5cdf7d09ab747442f08ea4, NAME => 'testSplitTableRegion,,1681268148351.c426152f6c5cdf7d09ab747442f08ea4.', STARTKEY => '', ENDKEY => ''} 2023-04-12 02:55:48,728 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(174): System coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro loaded, priority=536870911. 2023-04-12 02:55:48,728 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testSplitTableRegion c426152f6c5cdf7d09ab747442f08ea4 2023-04-12 02:55:48,728 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated testSplitTableRegion,,1681268148351.c426152f6c5cdf7d09ab747442f08ea4.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-12 02:55:48,728 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for c426152f6c5cdf7d09ab747442f08ea4 2023-04-12 02:55:48,728 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for c426152f6c5cdf7d09ab747442f08ea4 2023-04-12 02:55:48,732 INFO [StoreOpener-c426152f6c5cdf7d09ab747442f08ea4-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf1 of region c426152f6c5cdf7d09ab747442f08ea4 2023-04-12 02:55:48,734 DEBUG [StoreOpener-c426152f6c5cdf7d09ab747442f08ea4-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/cf1 2023-04-12 02:55:48,734 DEBUG [StoreOpener-c426152f6c5cdf7d09ab747442f08ea4-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/cf1 2023-04-12 02:55:48,734 INFO [StoreOpener-c426152f6c5cdf7d09ab747442f08ea4-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region c426152f6c5cdf7d09ab747442f08ea4 columnFamilyName cf1 2023-04-12 02:55:48,734 INFO [StoreOpener-c426152f6c5cdf7d09ab747442f08ea4-1] regionserver.HStore(310): Store=c426152f6c5cdf7d09ab747442f08ea4/cf1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-12 02:55:48,735 INFO [StoreOpener-c426152f6c5cdf7d09ab747442f08ea4-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf2 of region c426152f6c5cdf7d09ab747442f08ea4 2023-04-12 02:55:48,736 DEBUG [StoreOpener-c426152f6c5cdf7d09ab747442f08ea4-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/cf2 2023-04-12 02:55:48,736 DEBUG [StoreOpener-c426152f6c5cdf7d09ab747442f08ea4-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/cf2 2023-04-12 02:55:48,736 INFO [StoreOpener-c426152f6c5cdf7d09ab747442f08ea4-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region c426152f6c5cdf7d09ab747442f08ea4 columnFamilyName cf2 2023-04-12 02:55:48,737 INFO [StoreOpener-c426152f6c5cdf7d09ab747442f08ea4-1] regionserver.HStore(310): Store=c426152f6c5cdf7d09ab747442f08ea4/cf2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-12 02:55:48,738 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4 2023-04-12 02:55:48,738 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4 2023-04-12 02:55:48,740 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testSplitTableRegion descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-04-12 02:55:48,742 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for c426152f6c5cdf7d09ab747442f08ea4 2023-04-12 02:55:48,743 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-12 02:55:48,744 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened c426152f6c5cdf7d09ab747442f08ea4; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=62044765, jitterRate=-0.07546095550060272}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-04-12 02:55:48,744 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for c426152f6c5cdf7d09ab747442f08ea4: 2023-04-12 02:55:48,745 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testSplitTableRegion,,1681268148351.c426152f6c5cdf7d09ab747442f08ea4., pid=143, masterSystemTime=1681268148722 2023-04-12 02:55:48,747 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testSplitTableRegion,,1681268148351.c426152f6c5cdf7d09ab747442f08ea4. 2023-04-12 02:55:48,747 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testSplitTableRegion,,1681268148351.c426152f6c5cdf7d09ab747442f08ea4. 2023-04-12 02:55:48,747 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=142 updating hbase:meta row=c426152f6c5cdf7d09ab747442f08ea4, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:48,747 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testSplitTableRegion,,1681268148351.c426152f6c5cdf7d09ab747442f08ea4.","families":{"info":[{"qualifier":"regioninfo","vlen":54,"tag":[],"timestamp":"1681268148747"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1681268148747"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681268148747"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681268148747"}]},"ts":"1681268148747"} 2023-04-12 02:55:48,755 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=143, resume processing ppid=142 2023-04-12 02:55:48,755 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=143, ppid=142, state=SUCCESS; OpenRegionProcedure c426152f6c5cdf7d09ab747442f08ea4, server=jenkins-hbase4.apache.org,34579,1681268031719 in 181 msec 2023-04-12 02:55:48,757 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=142, resume processing ppid=141 2023-04-12 02:55:48,757 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=142, ppid=141, state=SUCCESS; TransitRegionStateProcedure table=testSplitTableRegion, region=c426152f6c5cdf7d09ab747442f08ea4, ASSIGN in 339 msec 2023-04-12 02:55:48,758 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=141, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=testSplitTableRegion execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-12 02:55:48,758 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testSplitTableRegion","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681268148758"}]},"ts":"1681268148758"} 2023-04-12 02:55:48,759 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testSplitTableRegion, state=ENABLED in hbase:meta 2023-04-12 02:55:48,762 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=141, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=testSplitTableRegion execute state=CREATE_TABLE_POST_OPERATION 2023-04-12 02:55:48,766 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=141, state=SUCCESS; CreateTableProcedure table=testSplitTableRegion in 411 msec 2023-04-12 02:55:48,860 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 02:55:48,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] procedure.ProcedureCoordinator(165): Submitting procedure testSplitTableRegion 2023-04-12 02:55:48,862 INFO [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testSplitTableRegion' 2023-04-12 02:55:48,862 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:55:48,862 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testSplitTableRegion' starting 'acquire' 2023-04-12 02:55:48,862 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testSplitTableRegion', kicking off acquire phase on members. 2023-04-12 02:55:48,862 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-12 02:55:48,862 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testSplitTableRegion 2023-04-12 02:55:48,864 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:55:48,864 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:48,865 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:55:48,865 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:55:48,864 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:55:48,865 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:55:48,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:55:48,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:55:48,865 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:55:48,865 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:48,865 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 02:55:48,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:55:48,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegion 2023-04-12 02:55:48,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegion 2023-04-12 02:55:48,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegion 2023-04-12 02:55:48,865 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-12 02:55:48,866 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-12 02:55:48,866 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-12 02:55:48,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 02:55:48,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegion 2023-04-12 02:55:48,866 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegion 2023-04-12 02:55:48,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 02:55:48,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegion 2023-04-12 02:55:48,866 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegion 2023-04-12 02:55:48,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 02:55:48,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegion 2023-04-12 02:55:48,866 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegion 2023-04-12 02:55:48,866 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegion 2023-04-12 02:55:48,866 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegion' with timeout 60000ms 2023-04-12 02:55:48,866 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:55:48,866 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegion 2023-04-12 02:55:48,867 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegion' starting 'acquire' stage 2023-04-12 02:55:48,867 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegion' locally acquired 2023-04-12 02:55:48,867 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44679,1681268031772' joining acquired barrier for procedure (testSplitTableRegion) in zk 2023-04-12 02:55:48,866 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegion' with timeout 60000ms 2023-04-12 02:55:48,867 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:55:48,867 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegion 2023-04-12 02:55:48,867 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegion' starting 'acquire' stage 2023-04-12 02:55:48,867 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegion' with timeout 60000ms 2023-04-12 02:55:48,867 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:55:48,867 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegion' starting 'acquire' stage 2023-04-12 02:55:48,867 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegion' locally acquired 2023-04-12 02:55:48,867 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43091,1681268031804' joining acquired barrier for procedure (testSplitTableRegion) in zk 2023-04-12 02:55:48,867 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 02:55:48,867 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 02:55:48,867 DEBUG [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testSplitTableRegion,,1681268148351.c426152f6c5cdf7d09ab747442f08ea4. 2023-04-12 02:55:48,867 DEBUG [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testSplitTableRegion,,1681268148351.c426152f6c5cdf7d09ab747442f08ea4. started... 2023-04-12 02:55:48,869 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-12 02:55:48,869 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-12 02:55:48,869 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegion' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 02:55:48,870 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-12 02:55:48,870 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-12 02:55:48,870 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegion' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 02:55:48,873 INFO [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c426152f6c5cdf7d09ab747442f08ea4 2/2 column families, dataSize=120 B heapSize=928 B 2023-04-12 02:55:48,887 INFO [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=60 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/.tmp/cf1/bd8f39350eae4e18bebd947afb5d9f5e 2023-04-12 02:55:48,908 INFO [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=60 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/.tmp/cf2/ba426c6436064f9ca5ca16cecc0d8c98 2023-04-12 02:55:48,912 DEBUG [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/.tmp/cf1/bd8f39350eae4e18bebd947afb5d9f5e as hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/cf1/bd8f39350eae4e18bebd947afb5d9f5e 2023-04-12 02:55:48,916 INFO [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/cf1/bd8f39350eae4e18bebd947afb5d9f5e, entries=2, sequenceid=6, filesize=4.8 K 2023-04-12 02:55:48,917 DEBUG [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/.tmp/cf2/ba426c6436064f9ca5ca16cecc0d8c98 as hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/cf2/ba426c6436064f9ca5ca16cecc0d8c98 2023-04-12 02:55:48,921 INFO [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/cf2/ba426c6436064f9ca5ca16cecc0d8c98, entries=2, sequenceid=6, filesize=4.8 K 2023-04-12 02:55:48,922 INFO [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~120 B/120, heapSize ~896 B/896, currentSize=0 B/0 for c426152f6c5cdf7d09ab747442f08ea4 in 49ms, sequenceid=6, compaction requested=false 2023-04-12 02:55:48,922 DEBUG [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'testSplitTableRegion' 2023-04-12 02:55:48,923 DEBUG [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c426152f6c5cdf7d09ab747442f08ea4: 2023-04-12 02:55:48,923 DEBUG [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testSplitTableRegion,,1681268148351.c426152f6c5cdf7d09ab747442f08ea4. 2023-04-12 02:55:48,923 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 02:55:48,923 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 02:55:48,923 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:48,923 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegion' locally acquired 2023-04-12 02:55:48,923 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34579,1681268031719' joining acquired barrier for procedure (testSplitTableRegion) in zk 2023-04-12 02:55:48,924 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:48,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:48,925 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-12 02:55:48,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 02:55:48,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 02:55:48,925 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-12 02:55:48,925 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegion' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 02:55:48,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 02:55:48,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 02:55:48,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegion 2023-04-12 02:55:48,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:48,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:48,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:48,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 02:55:48,927 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34579,1681268031719' joining acquired barrier for procedure 'testSplitTableRegion' on coordinator 2023-04-12 02:55:48,927 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@79497fc1[Count = 0] remaining members to acquire global barrier 2023-04-12 02:55:48,927 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testSplitTableRegion' starting 'in-barrier' execution. 2023-04-12 02:55:48,927 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-12 02:55:48,928 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-12 02:55:48,928 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-12 02:55:48,928 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-12 02:55:48,928 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-12 02:55:48,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-12 02:55:48,928 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-12 02:55:48,928 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegion' received 'reached' from coordinator. 2023-04-12 02:55:48,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-12 02:55:48,928 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-12 02:55:48,929 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:48,929 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 02:55:48,929 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegion' locally completed 2023-04-12 02:55:48,929 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegion' received 'reached' from coordinator. 2023-04-12 02:55:48,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-12 02:55:48,929 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegion' locally completed 2023-04-12 02:55:48,929 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegion' completed for member 'jenkins-hbase4.apache.org,44679,1681268031772' in zk 2023-04-12 02:55:48,929 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegion' completed for member 'jenkins-hbase4.apache.org,34579,1681268031719' in zk 2023-04-12 02:55:48,929 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegion' received 'reached' from coordinator. 2023-04-12 02:55:48,929 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegion' locally completed 2023-04-12 02:55:48,929 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegion' completed for member 'jenkins-hbase4.apache.org,43091,1681268031804' in zk 2023-04-12 02:55:48,931 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegion' has notified controller of completion 2023-04-12 02:55:48,931 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:55:48,931 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegion' completed. 2023-04-12 02:55:48,932 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegion' has notified controller of completion 2023-04-12 02:55:48,932 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:55:48,932 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegion' completed. 2023-04-12 02:55:48,936 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegion' has notified controller of completion 2023-04-12 02:55:48,936 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:55:48,936 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegion' completed. 2023-04-12 02:55:48,936 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:48,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:48,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 02:55:48,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 02:55:48,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 02:55:48,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 02:55:48,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegion 2023-04-12 02:55:48,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:48,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:48,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:48,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 02:55:48,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegion 2023-04-12 02:55:48,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:48,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:48,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:48,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testSplitTableRegion' member 'jenkins-hbase4.apache.org,34579,1681268031719': 2023-04-12 02:55:48,942 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34579,1681268031719' released barrier for procedure'testSplitTableRegion', counting down latch. Waiting for 0 more 2023-04-12 02:55:48,942 INFO [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testSplitTableRegion' execution completed 2023-04-12 02:55:48,942 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 02:55:48,942 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 02:55:48,942 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testSplitTableRegion 2023-04-12 02:55:48,942 INFO [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testSplitTableRegionincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 02:55:48,943 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-12 02:55:48,943 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-12 02:55:48,944 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-12 02:55:48,944 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-12 02:55:48,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-12 02:55:48,943 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-12 02:55:48,944 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:55:48,944 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-12 02:55:48,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-12 02:55:48,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-12 02:55:48,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 02:55:48,944 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-12 02:55:48,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-12 02:55:48,944 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:55:48,944 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:55:48,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:55:48,944 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:48,944 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:55:48,944 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:55:48,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:55:48,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 02:55:48,945 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:55:48,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:55:48,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-12 02:55:48,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-12 02:55:48,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-12 02:55:48,946 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:48,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 02:55:48,947 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:48,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegion 2023-04-12 02:55:48,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 02:55:48,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegion 2023-04-12 02:55:48,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:48,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:48,949 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:48,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:48,950 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:48,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 02:55:48,950 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:48,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegion 2023-04-12 02:55:48,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:48,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:48,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:48,954 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:48,954 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:55:48,954 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:55:48,954 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:55:48,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:55:48,954 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:55:48,954 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:55:48,954 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegion 2023-04-12 02:55:48,954 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:55:48,954 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:48,954 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:55:48,954 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:55:48,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:55:48,954 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:48,954 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegion 2023-04-12 02:55:48,954 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:48,954 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:55:48,954 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-12 02:55:48,954 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:48,954 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:48,954 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-12 02:55:48,955 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-12 02:55:48,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:55:48,955 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:55:48,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:55:48,955 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:55:48,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:55:48,955 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:55:48,955 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testSplitTableRegion' 2023-04-12 02:55:48,956 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 02:55:48,956 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:55:48,956 DEBUG [Listener at localhost/46667] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testSplitTableRegion'' to complete. (max 20000 ms per retry) 2023-04-12 02:55:48,956 DEBUG [Listener at localhost/46667] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 02:55:48,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:55:49,056 DEBUG [Listener at localhost/46667] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 02:55:49,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 02:55:49,069 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 02:55:49,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] procedure.ProcedureCoordinator(143): Procedure testSplitTableRegion was in running list but was completed. Accepting new attempt. 2023-04-12 02:55:49,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] procedure.ProcedureCoordinator(165): Submitting procedure testSplitTableRegion 2023-04-12 02:55:49,072 INFO [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testSplitTableRegion' 2023-04-12 02:55:49,072 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:55:49,072 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testSplitTableRegion' starting 'acquire' 2023-04-12 02:55:49,072 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testSplitTableRegion', kicking off acquire phase on members. 2023-04-12 02:55:49,072 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-12 02:55:49,072 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testSplitTableRegion 2023-04-12 02:55:49,074 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:55:49,074 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:55:49,074 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:55:49,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:55:49,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:55:49,074 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:49,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:55:49,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:55:49,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:55:49,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:55:49,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegion 2023-04-12 02:55:49,075 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:49,075 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 02:55:49,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegion 2023-04-12 02:55:49,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegion 2023-04-12 02:55:49,076 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-12 02:55:49,076 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-12 02:55:49,076 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-12 02:55:49,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 02:55:49,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegion 2023-04-12 02:55:49,076 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegion 2023-04-12 02:55:49,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 02:55:49,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegion 2023-04-12 02:55:49,076 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegion 2023-04-12 02:55:49,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 02:55:49,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegion 2023-04-12 02:55:49,077 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegion 2023-04-12 02:55:49,077 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegion is still present, removing 2023-04-12 02:55:49,077 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegion 2023-04-12 02:55:49,077 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegion' with timeout 60000ms 2023-04-12 02:55:49,077 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:55:49,077 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegion' starting 'acquire' stage 2023-04-12 02:55:49,077 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegion' locally acquired 2023-04-12 02:55:49,077 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44679,1681268031772' joining acquired barrier for procedure (testSplitTableRegion) in zk 2023-04-12 02:55:49,078 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-12 02:55:49,079 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-12 02:55:49,079 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegion' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 02:55:49,080 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegion is still present, removing 2023-04-12 02:55:49,080 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegion 2023-04-12 02:55:49,080 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegion' with timeout 60000ms 2023-04-12 02:55:49,080 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:55:49,080 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegion' starting 'acquire' stage 2023-04-12 02:55:49,084 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegion is still present, removing 2023-04-12 02:55:49,084 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegion 2023-04-12 02:55:49,084 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegion' with timeout 60000ms 2023-04-12 02:55:49,084 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:55:49,084 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegion' starting 'acquire' stage 2023-04-12 02:55:49,084 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegion' locally acquired 2023-04-12 02:55:49,084 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43091,1681268031804' joining acquired barrier for procedure (testSplitTableRegion) in zk 2023-04-12 02:55:49,085 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 02:55:49,085 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 02:55:49,085 DEBUG [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testSplitTableRegion,,1681268148351.c426152f6c5cdf7d09ab747442f08ea4. 2023-04-12 02:55:49,085 DEBUG [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testSplitTableRegion,,1681268148351.c426152f6c5cdf7d09ab747442f08ea4. started... 2023-04-12 02:55:49,085 INFO [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c426152f6c5cdf7d09ab747442f08ea4 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-04-12 02:55:49,085 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-12 02:55:49,087 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-12 02:55:49,087 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegion' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 02:55:49,517 INFO [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=19 (bloomFilter=true), to=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/.tmp/cf1/72636f8bfb49470784b49ea8615a5c6a 2023-04-12 02:55:49,530 INFO [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=19 (bloomFilter=true), to=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/.tmp/cf2/f5ea9d0c0a4c491f91c40210e3a13892 2023-04-12 02:55:49,534 DEBUG [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/.tmp/cf1/72636f8bfb49470784b49ea8615a5c6a as hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/cf1/72636f8bfb49470784b49ea8615a5c6a 2023-04-12 02:55:49,537 INFO [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/cf1/72636f8bfb49470784b49ea8615a5c6a, entries=10, sequenceid=19, filesize=5.0 K 2023-04-12 02:55:49,538 DEBUG [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/.tmp/cf2/f5ea9d0c0a4c491f91c40210e3a13892 as hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/cf2/f5ea9d0c0a4c491f91c40210e3a13892 2023-04-12 02:55:49,542 INFO [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/cf2/f5ea9d0c0a4c491f91c40210e3a13892, entries=10, sequenceid=19, filesize=5.0 K 2023-04-12 02:55:49,543 INFO [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for c426152f6c5cdf7d09ab747442f08ea4 in 458ms, sequenceid=19, compaction requested=false 2023-04-12 02:55:49,543 DEBUG [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c426152f6c5cdf7d09ab747442f08ea4: 2023-04-12 02:55:49,543 DEBUG [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testSplitTableRegion,,1681268148351.c426152f6c5cdf7d09ab747442f08ea4. 2023-04-12 02:55:49,544 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 02:55:49,544 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 02:55:49,544 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:49,544 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegion' locally acquired 2023-04-12 02:55:49,544 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34579,1681268031719' joining acquired barrier for procedure (testSplitTableRegion) in zk 2023-04-12 02:55:49,547 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:49,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:49,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 02:55:49,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 02:55:49,547 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-12 02:55:49,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 02:55:49,547 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-12 02:55:49,547 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegion' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 02:55:49,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 02:55:49,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegion 2023-04-12 02:55:49,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:49,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:49,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:49,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 02:55:49,549 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34579,1681268031719' joining acquired barrier for procedure 'testSplitTableRegion' on coordinator 2023-04-12 02:55:49,549 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6a0915fa[Count = 0] remaining members to acquire global barrier 2023-04-12 02:55:49,549 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testSplitTableRegion' starting 'in-barrier' execution. 2023-04-12 02:55:49,550 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-12 02:55:49,551 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-12 02:55:49,551 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-12 02:55:49,551 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-12 02:55:49,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-12 02:55:49,551 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegion' received 'reached' from coordinator. 2023-04-12 02:55:49,551 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-12 02:55:49,551 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegion' locally completed 2023-04-12 02:55:49,551 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-12 02:55:49,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-12 02:55:49,551 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegion' completed for member 'jenkins-hbase4.apache.org,34579,1681268031719' in zk 2023-04-12 02:55:49,551 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:49,551 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 02:55:49,551 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-12 02:55:49,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-12 02:55:49,551 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegion' received 'reached' from coordinator. 2023-04-12 02:55:49,551 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegion' received 'reached' from coordinator. 2023-04-12 02:55:49,551 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegion' locally completed 2023-04-12 02:55:49,552 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegion' completed for member 'jenkins-hbase4.apache.org,43091,1681268031804' in zk 2023-04-12 02:55:49,551 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegion' locally completed 2023-04-12 02:55:49,552 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegion' completed for member 'jenkins-hbase4.apache.org,44679,1681268031772' in zk 2023-04-12 02:55:49,552 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:49,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:49,552 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegion' has notified controller of completion 2023-04-12 02:55:49,553 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:55:49,553 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegion' completed. 2023-04-12 02:55:49,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 02:55:49,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 02:55:49,553 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegion' has notified controller of completion 2023-04-12 02:55:49,553 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:55:49,553 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegion' completed. 2023-04-12 02:55:49,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 02:55:49,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 02:55:49,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegion 2023-04-12 02:55:49,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:49,554 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegion' has notified controller of completion 2023-04-12 02:55:49,554 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:55:49,554 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegion' completed. 2023-04-12 02:55:49,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:49,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:49,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 02:55:49,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegion 2023-04-12 02:55:49,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:49,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:49,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:49,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testSplitTableRegion' member 'jenkins-hbase4.apache.org,34579,1681268031719': 2023-04-12 02:55:49,558 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34579,1681268031719' released barrier for procedure'testSplitTableRegion', counting down latch. Waiting for 0 more 2023-04-12 02:55:49,558 INFO [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testSplitTableRegion' execution completed 2023-04-12 02:55:49,558 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 02:55:49,558 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 02:55:49,558 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testSplitTableRegion 2023-04-12 02:55:49,558 INFO [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testSplitTableRegionincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 02:55:49,561 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-12 02:55:49,561 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-12 02:55:49,562 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-12 02:55:49,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-12 02:55:49,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-12 02:55:49,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-12 02:55:49,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-12 02:55:49,562 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-12 02:55:49,562 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:55:49,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-12 02:55:49,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-12 02:55:49,562 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:55:49,562 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:55:49,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-12 02:55:49,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 02:55:49,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 02:55:49,563 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:55:49,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 02:55:49,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:55:49,563 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:49,563 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:55:49,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:55:49,563 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:55:49,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:55:49,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-12 02:55:49,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegion 2023-04-12 02:55:49,564 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:49,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-12 02:55:49,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-12 02:55:49,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 02:55:49,564 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:49,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegion 2023-04-12 02:55:49,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:49,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:49,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:49,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 02:55:49,567 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:49,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegion 2023-04-12 02:55:49,567 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:49,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:49,568 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:49,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:49,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:49,570 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:49,570 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegion 2023-04-12 02:55:49,570 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:49,571 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:49,571 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegion 2023-04-12 02:55:49,571 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:49,571 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-12 02:55:49,571 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:49,571 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:49,571 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:55:49,571 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-12 02:55:49,571 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-12 02:55:49,571 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testSplitTableRegion' 2023-04-12 02:55:49,571 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 02:55:49,571 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:55:49,571 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:55:49,571 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:55:49,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:55:49,571 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:55:49,571 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:55:49,572 DEBUG [Listener at localhost/46667] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testSplitTableRegion'' to complete. (max 20000 ms per retry) 2023-04-12 02:55:49,572 DEBUG [Listener at localhost/46667] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 02:55:49,572 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:55:49,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:55:49,572 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:55:49,572 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:55:49,572 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:55:49,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:55:49,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:55:49,572 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:55:49,572 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:55:49,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:55:49,573 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:55:49,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:55:49,672 DEBUG [Listener at localhost/46667] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 02:55:49,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 02:55:49,688 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 02:55:49,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] procedure.ProcedureCoordinator(143): Procedure testSplitTableRegion was in running list but was completed. Accepting new attempt. 2023-04-12 02:55:49,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] procedure.ProcedureCoordinator(165): Submitting procedure testSplitTableRegion 2023-04-12 02:55:49,689 INFO [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testSplitTableRegion' 2023-04-12 02:55:49,689 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:55:49,689 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testSplitTableRegion' starting 'acquire' 2023-04-12 02:55:49,689 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testSplitTableRegion', kicking off acquire phase on members. 2023-04-12 02:55:49,690 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-12 02:55:49,690 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testSplitTableRegion 2023-04-12 02:55:49,691 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:55:49,691 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:55:49,691 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:55:49,691 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:55:49,691 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:49,692 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:55:49,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:55:49,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:55:49,691 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:55:49,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:55:49,692 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:49,692 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 02:55:49,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegion 2023-04-12 02:55:49,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegion 2023-04-12 02:55:49,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegion 2023-04-12 02:55:49,692 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-12 02:55:49,692 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-12 02:55:49,693 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-12 02:55:49,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 02:55:49,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegion 2023-04-12 02:55:49,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 02:55:49,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 02:55:49,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegion 2023-04-12 02:55:49,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegion 2023-04-12 02:55:49,693 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegion 2023-04-12 02:55:49,693 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegion 2023-04-12 02:55:49,693 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegion 2023-04-12 02:55:49,693 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegion is still present, removing 2023-04-12 02:55:49,693 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegion 2023-04-12 02:55:49,693 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegion' with timeout 60000ms 2023-04-12 02:55:49,693 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:55:49,693 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegion is still present, removing 2023-04-12 02:55:49,693 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegion' starting 'acquire' stage 2023-04-12 02:55:49,693 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegion is still present, removing 2023-04-12 02:55:49,693 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegion 2023-04-12 02:55:49,693 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegion 2023-04-12 02:55:49,694 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegion' with timeout 60000ms 2023-04-12 02:55:49,694 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:55:49,694 DEBUG [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testSplitTableRegion,,1681268148351.c426152f6c5cdf7d09ab747442f08ea4. 2023-04-12 02:55:49,694 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegion' with timeout 60000ms 2023-04-12 02:55:49,694 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:55:49,694 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 02:55:49,694 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 02:55:49,694 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegion' starting 'acquire' stage 2023-04-12 02:55:49,694 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegion' locally acquired 2023-04-12 02:55:49,694 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43091,1681268031804' joining acquired barrier for procedure (testSplitTableRegion) in zk 2023-04-12 02:55:49,694 DEBUG [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testSplitTableRegion,,1681268148351.c426152f6c5cdf7d09ab747442f08ea4. started... 2023-04-12 02:55:49,694 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegion' starting 'acquire' stage 2023-04-12 02:55:49,694 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegion' locally acquired 2023-04-12 02:55:49,694 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44679,1681268031772' joining acquired barrier for procedure (testSplitTableRegion) in zk 2023-04-12 02:55:49,694 INFO [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c426152f6c5cdf7d09ab747442f08ea4 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-04-12 02:55:49,696 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-12 02:55:49,699 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-12 02:55:49,700 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-12 02:55:49,700 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegion' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 02:55:49,700 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-12 02:55:49,700 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegion' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 02:55:49,708 INFO [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=32 (bloomFilter=true), to=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/.tmp/cf1/3e014fee55b841d9a93f29d49aa55156 2023-04-12 02:55:49,726 INFO [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=32 (bloomFilter=true), to=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/.tmp/cf2/c4a22b15c4714bc1a6f7b206d3315186 2023-04-12 02:55:49,730 DEBUG [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/.tmp/cf1/3e014fee55b841d9a93f29d49aa55156 as hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/cf1/3e014fee55b841d9a93f29d49aa55156 2023-04-12 02:55:49,734 INFO [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/cf1/3e014fee55b841d9a93f29d49aa55156, entries=10, sequenceid=32, filesize=5.0 K 2023-04-12 02:55:49,735 DEBUG [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/.tmp/cf2/c4a22b15c4714bc1a6f7b206d3315186 as hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/cf2/c4a22b15c4714bc1a6f7b206d3315186 2023-04-12 02:55:49,738 INFO [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/cf2/c4a22b15c4714bc1a6f7b206d3315186, entries=10, sequenceid=32, filesize=5.0 K 2023-04-12 02:55:49,739 INFO [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for c426152f6c5cdf7d09ab747442f08ea4 in 45ms, sequenceid=32, compaction requested=true 2023-04-12 02:55:49,739 DEBUG [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c426152f6c5cdf7d09ab747442f08ea4: 2023-04-12 02:55:49,739 DEBUG [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testSplitTableRegion,,1681268148351.c426152f6c5cdf7d09ab747442f08ea4. 2023-04-12 02:55:49,739 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 02:55:49,739 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 02:55:49,739 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:49,739 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegion' locally acquired 2023-04-12 02:55:49,739 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34579,1681268031719' joining acquired barrier for procedure (testSplitTableRegion) in zk 2023-04-12 02:55:49,741 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:49,741 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-12 02:55:49,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:49,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 02:55:49,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 02:55:49,741 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-12 02:55:49,741 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegion' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 02:55:49,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 02:55:49,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 02:55:49,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegion 2023-04-12 02:55:49,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:49,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:49,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:49,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 02:55:49,743 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34579,1681268031719' joining acquired barrier for procedure 'testSplitTableRegion' on coordinator 2023-04-12 02:55:49,743 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@61706e13[Count = 0] remaining members to acquire global barrier 2023-04-12 02:55:49,743 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testSplitTableRegion' starting 'in-barrier' execution. 2023-04-12 02:55:49,743 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-12 02:55:49,745 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-12 02:55:49,745 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-12 02:55:49,745 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-12 02:55:49,745 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-12 02:55:49,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-12 02:55:49,745 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-12 02:55:49,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-12 02:55:49,746 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegion' received 'reached' from coordinator. 2023-04-12 02:55:49,746 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:49,746 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 02:55:49,745 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-12 02:55:49,746 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegion' received 'reached' from coordinator. 2023-04-12 02:55:49,746 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegion' locally completed 2023-04-12 02:55:49,746 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegion' locally completed 2023-04-12 02:55:49,746 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegion' completed for member 'jenkins-hbase4.apache.org,34579,1681268031719' in zk 2023-04-12 02:55:49,746 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegion' completed for member 'jenkins-hbase4.apache.org,43091,1681268031804' in zk 2023-04-12 02:55:49,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-12 02:55:49,746 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegion' received 'reached' from coordinator. 2023-04-12 02:55:49,746 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegion' locally completed 2023-04-12 02:55:49,746 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegion' completed for member 'jenkins-hbase4.apache.org,44679,1681268031772' in zk 2023-04-12 02:55:49,747 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegion' has notified controller of completion 2023-04-12 02:55:49,747 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:49,748 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:55:49,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:49,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 02:55:49,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 02:55:49,748 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegion' completed. 2023-04-12 02:55:49,748 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegion' has notified controller of completion 2023-04-12 02:55:49,748 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:55:49,748 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegion' completed. 2023-04-12 02:55:49,749 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegion' has notified controller of completion 2023-04-12 02:55:49,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 02:55:49,749 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:55:49,749 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegion' completed. 2023-04-12 02:55:49,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 02:55:49,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegion 2023-04-12 02:55:49,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:49,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:49,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:49,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 02:55:49,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegion 2023-04-12 02:55:49,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:49,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:49,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:49,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testSplitTableRegion' member 'jenkins-hbase4.apache.org,34579,1681268031719': 2023-04-12 02:55:49,752 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34579,1681268031719' released barrier for procedure'testSplitTableRegion', counting down latch. Waiting for 0 more 2023-04-12 02:55:49,752 INFO [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testSplitTableRegion' execution completed 2023-04-12 02:55:49,752 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 02:55:49,752 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 02:55:49,752 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testSplitTableRegion 2023-04-12 02:55:49,752 INFO [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testSplitTableRegionincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 02:55:49,755 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-12 02:55:49,755 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-12 02:55:49,755 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-12 02:55:49,755 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-12 02:55:49,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-12 02:55:49,755 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-12 02:55:49,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-12 02:55:49,755 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:55:49,755 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-12 02:55:49,755 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:55:49,755 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:55:49,755 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-12 02:55:49,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-12 02:55:49,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-12 02:55:49,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 02:55:49,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 02:55:49,755 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:55:49,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:55:49,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-12 02:55:49,756 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:49,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 02:55:49,756 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:55:49,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:55:49,756 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:55:49,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:55:49,757 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:49,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegion 2023-04-12 02:55:49,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-12 02:55:49,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-12 02:55:49,757 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:49,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 02:55:49,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegion 2023-04-12 02:55:49,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:49,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:49,759 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:49,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:49,759 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:49,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 02:55:49,759 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:49,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegion 2023-04-12 02:55:49,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:49,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:49,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:49,762 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:55:49,762 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:55:49,762 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:55:49,762 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:55:49,762 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:49,762 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:55:49,762 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:55:49,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:55:49,762 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:55:49,762 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegion 2023-04-12 02:55:49,762 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:55:49,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:55:49,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:55:49,762 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testSplitTableRegion' 2023-04-12 02:55:49,762 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 02:55:49,762 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:55:49,762 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:49,763 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:49,763 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegion 2023-04-12 02:55:49,763 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:49,763 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-12 02:55:49,763 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:49,763 DEBUG [Listener at localhost/46667] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testSplitTableRegion'' to complete. (max 20000 ms per retry) 2023-04-12 02:55:49,762 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:55:49,763 DEBUG [Listener at localhost/46667] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 02:55:49,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:55:49,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:55:49,763 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:49,762 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:55:49,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:55:49,763 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-12 02:55:49,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:55:49,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:55:49,763 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-12 02:55:49,863 DEBUG [Listener at localhost/46667] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 02:55:49,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 02:55:49,876 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 02:55:49,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] procedure.ProcedureCoordinator(143): Procedure testSplitTableRegion was in running list but was completed. Accepting new attempt. 2023-04-12 02:55:49,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] procedure.ProcedureCoordinator(165): Submitting procedure testSplitTableRegion 2023-04-12 02:55:49,877 INFO [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testSplitTableRegion' 2023-04-12 02:55:49,877 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:55:49,878 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testSplitTableRegion' starting 'acquire' 2023-04-12 02:55:49,878 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testSplitTableRegion', kicking off acquire phase on members. 2023-04-12 02:55:49,878 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-12 02:55:49,878 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testSplitTableRegion 2023-04-12 02:55:49,879 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:55:49,879 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:55:49,879 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:49,879 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:55:49,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:55:49,879 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:55:49,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:55:49,879 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:55:49,879 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:49,880 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 02:55:49,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:55:49,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:55:49,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegion 2023-04-12 02:55:49,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegion 2023-04-12 02:55:49,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegion 2023-04-12 02:55:49,880 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-12 02:55:49,880 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-12 02:55:49,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 02:55:49,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 02:55:49,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegion 2023-04-12 02:55:49,880 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-12 02:55:49,880 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegion 2023-04-12 02:55:49,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegion 2023-04-12 02:55:49,881 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegion 2023-04-12 02:55:49,881 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegion is still present, removing 2023-04-12 02:55:49,881 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegion 2023-04-12 02:55:49,881 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegion is still present, removing 2023-04-12 02:55:49,881 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegion' with timeout 60000ms 2023-04-12 02:55:49,881 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:55:49,881 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegion' starting 'acquire' stage 2023-04-12 02:55:49,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 02:55:49,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegion 2023-04-12 02:55:49,881 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegion 2023-04-12 02:55:49,881 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 02:55:49,881 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 02:55:49,881 DEBUG [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testSplitTableRegion,,1681268148351.c426152f6c5cdf7d09ab747442f08ea4. 2023-04-12 02:55:49,881 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegion is still present, removing 2023-04-12 02:55:49,881 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegion 2023-04-12 02:55:49,881 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegion 2023-04-12 02:55:49,881 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegion' with timeout 60000ms 2023-04-12 02:55:49,881 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegion' with timeout 60000ms 2023-04-12 02:55:49,882 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:55:49,881 DEBUG [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testSplitTableRegion,,1681268148351.c426152f6c5cdf7d09ab747442f08ea4. started... 2023-04-12 02:55:49,882 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegion' starting 'acquire' stage 2023-04-12 02:55:49,882 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegion' locally acquired 2023-04-12 02:55:49,881 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:55:49,882 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44679,1681268031772' joining acquired barrier for procedure (testSplitTableRegion) in zk 2023-04-12 02:55:49,882 INFO [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c426152f6c5cdf7d09ab747442f08ea4 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-04-12 02:55:49,882 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegion' starting 'acquire' stage 2023-04-12 02:55:49,882 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegion' locally acquired 2023-04-12 02:55:49,882 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43091,1681268031804' joining acquired barrier for procedure (testSplitTableRegion) in zk 2023-04-12 02:55:49,883 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-12 02:55:49,888 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-12 02:55:49,888 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-12 02:55:49,888 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegion' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 02:55:49,888 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-12 02:55:49,888 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegion' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 02:55:49,898 INFO [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=45 (bloomFilter=true), to=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/.tmp/cf1/52c29c1aa57f4576b791c97686dd2e0d 2023-04-12 02:55:49,914 INFO [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=45 (bloomFilter=true), to=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/.tmp/cf2/e4c7205c2e024488b0db4154888f3f50 2023-04-12 02:55:49,918 DEBUG [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/.tmp/cf1/52c29c1aa57f4576b791c97686dd2e0d as hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/cf1/52c29c1aa57f4576b791c97686dd2e0d 2023-04-12 02:55:49,921 INFO [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/cf1/52c29c1aa57f4576b791c97686dd2e0d, entries=10, sequenceid=45, filesize=5.0 K 2023-04-12 02:55:49,922 DEBUG [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/.tmp/cf2/e4c7205c2e024488b0db4154888f3f50 as hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/cf2/e4c7205c2e024488b0db4154888f3f50 2023-04-12 02:55:49,925 INFO [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/cf2/e4c7205c2e024488b0db4154888f3f50, entries=10, sequenceid=45, filesize=5.0 K 2023-04-12 02:55:49,925 INFO [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for c426152f6c5cdf7d09ab747442f08ea4 in 43ms, sequenceid=45, compaction requested=true 2023-04-12 02:55:49,926 DEBUG [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c426152f6c5cdf7d09ab747442f08ea4: 2023-04-12 02:55:49,926 DEBUG [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testSplitTableRegion,,1681268148351.c426152f6c5cdf7d09ab747442f08ea4. 2023-04-12 02:55:49,926 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 02:55:49,926 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 02:55:49,926 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:49,926 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegion' locally acquired 2023-04-12 02:55:49,926 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34579,1681268031719' joining acquired barrier for procedure (testSplitTableRegion) in zk 2023-04-12 02:55:49,927 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-12 02:55:49,927 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:49,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:49,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 02:55:49,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 02:55:49,927 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-12 02:55:49,928 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegion' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 02:55:49,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 02:55:49,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 02:55:49,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegion 2023-04-12 02:55:49,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:49,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:49,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:49,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 02:55:49,929 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34579,1681268031719' joining acquired barrier for procedure 'testSplitTableRegion' on coordinator 2023-04-12 02:55:49,929 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@32a2d947[Count = 0] remaining members to acquire global barrier 2023-04-12 02:55:49,929 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testSplitTableRegion' starting 'in-barrier' execution. 2023-04-12 02:55:49,929 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-12 02:55:49,931 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-12 02:55:49,931 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-12 02:55:49,931 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-12 02:55:49,931 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-12 02:55:49,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-12 02:55:49,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-12 02:55:49,931 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegion' received 'reached' from coordinator. 2023-04-12 02:55:49,931 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegion' received 'reached' from coordinator. 2023-04-12 02:55:49,931 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegion' locally completed 2023-04-12 02:55:49,931 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:49,931 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 02:55:49,931 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegion' locally completed 2023-04-12 02:55:49,931 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegion' completed for member 'jenkins-hbase4.apache.org,34579,1681268031719' in zk 2023-04-12 02:55:49,931 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegion' completed for member 'jenkins-hbase4.apache.org,44679,1681268031772' in zk 2023-04-12 02:55:49,932 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegion' has notified controller of completion 2023-04-12 02:55:49,932 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:55:49,932 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegion' completed. 2023-04-12 02:55:49,934 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:49,934 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegion' has notified controller of completion 2023-04-12 02:55:49,934 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:55:49,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:49,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 02:55:49,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 02:55:49,934 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegion' completed. 2023-04-12 02:55:49,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 02:55:49,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 02:55:49,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegion 2023-04-12 02:55:49,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:49,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:49,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:49,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 02:55:49,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegion 2023-04-12 02:55:49,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:49,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:49,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testSplitTableRegion' member 'jenkins-hbase4.apache.org,34579,1681268031719': 2023-04-12 02:55:49,937 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34579,1681268031719' released barrier for procedure'testSplitTableRegion', counting down latch. Waiting for 0 more 2023-04-12 02:55:49,937 INFO [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testSplitTableRegion' execution completed 2023-04-12 02:55:49,937 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 02:55:49,937 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 02:55:49,937 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testSplitTableRegion 2023-04-12 02:55:49,937 INFO [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testSplitTableRegionincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 02:55:49,939 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-12 02:55:49,939 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-12 02:55:49,939 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-12 02:55:49,940 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-12 02:55:49,940 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-12 02:55:49,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-12 02:55:49,940 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-12 02:55:49,940 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:55:49,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-12 02:55:49,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 02:55:49,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 02:55:49,939 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:55:49,940 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-12 02:55:49,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-12 02:55:49,940 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-12 02:55:49,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-12 02:55:49,940 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:55:49,940 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-12 02:55:49,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-12 02:55:49,940 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegion' received 'reached' from coordinator. 2023-04-12 02:55:49,940 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegion' locally completed 2023-04-12 02:55:49,941 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegion' completed for member 'jenkins-hbase4.apache.org,43091,1681268031804' in zk 2023-04-12 02:55:49,941 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:55:49,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:55:49,941 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:55:49,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:55:49,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-12 02:55:49,942 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:55:49,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:55:49,942 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:49,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 02:55:49,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-12 02:55:49,943 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:49,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegion 2023-04-12 02:55:49,943 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:49,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 02:55:49,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegion 2023-04-12 02:55:49,945 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:49,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:49,945 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:49,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:49,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:49,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 02:55:49,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegion 2023-04-12 02:55:49,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:49,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-12 02:55:49,950 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:55:49,950 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:55:49,950 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/testSplitTableRegion because node does not exist (not an error) 2023-04-12 02:55:49,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:55:49,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:55:49,950 ERROR [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.ProcedureMember(208): Failed to post zk node:/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase4.apache.org,43091,1681268031804 to join procedure barrier. org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase4.apache.org,43091,1681268031804 at org.apache.zookeeper.KeeperException.create(KeeperException.java:118) at org.apache.zookeeper.KeeperException.create(KeeperException.java:54) at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:1538) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.createNonSequential(RecoverableZooKeeper.java:525) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.create(RecoverableZooKeeper.java:505) at org.apache.hadoop.hbase.zookeeper.ZKUtil.createAndFailSilent(ZKUtil.java:829) at org.apache.hadoop.hbase.zookeeper.ZKUtil.createAndFailSilent(ZKUtil.java:819) at org.apache.hadoop.hbase.procedure.ZKProcedureMemberRpcs.sendMemberCompleted(ZKProcedureMemberRpcs.java:276) at org.apache.hadoop.hbase.procedure.Subprocedure.call(Subprocedure.java:186) at org.apache.hadoop.hbase.procedure.Subprocedure.call(Subprocedure.java:46) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2023-04-12 02:55:49,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:49,951 ERROR [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(258): Failed to post zk node:/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase4.apache.org,43091,1681268031804 to join procedure barrier. org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase4.apache.org,43091,1681268031804 at org.apache.zookeeper.KeeperException.create(KeeperException.java:118) at org.apache.zookeeper.KeeperException.create(KeeperException.java:54) at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:1538) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.createNonSequential(RecoverableZooKeeper.java:525) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.create(RecoverableZooKeeper.java:505) at org.apache.hadoop.hbase.zookeeper.ZKUtil.createAndFailSilent(ZKUtil.java:829) at org.apache.hadoop.hbase.zookeeper.ZKUtil.createAndFailSilent(ZKUtil.java:819) at org.apache.hadoop.hbase.procedure.ZKProcedureMemberRpcs.sendMemberCompleted(ZKProcedureMemberRpcs.java:276) at org.apache.hadoop.hbase.procedure.Subprocedure.call(Subprocedure.java:186) at org.apache.hadoop.hbase.procedure.Subprocedure.call(Subprocedure.java:46) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2023-04-12 02:55:49,951 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:55:49,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:55:49,952 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/testSplitTableRegion because node does not exist (not an error) 2023-04-12 02:55:49,952 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] errorhandling.ForeignExceptionDispatcher(64): testSplitTableRegion accepting received exception org.apache.zookeeper.KeeperException$NoNodeException via jenkins-hbase4.apache.org,43091,1681268031804:org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase4.apache.org,43091,1681268031804 at org.apache.hadoop.hbase.procedure.Subprocedure.cancel(Subprocedure.java:263) at org.apache.hadoop.hbase.procedure.ProcedureMember.controllerConnectionFailure(ProcedureMember.java:214) at org.apache.hadoop.hbase.procedure.ZKProcedureMemberRpcs.sendMemberCompleted(ZKProcedureMemberRpcs.java:279) at org.apache.hadoop.hbase.procedure.Subprocedure.call(Subprocedure.java:186) at org.apache.hadoop.hbase.procedure.Subprocedure.call(Subprocedure.java:46) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase4.apache.org,43091,1681268031804 at org.apache.zookeeper.KeeperException.create(KeeperException.java:118) at org.apache.zookeeper.KeeperException.create(KeeperException.java:54) at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:1538) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.createNonSequential(RecoverableZooKeeper.java:525) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.create(RecoverableZooKeeper.java:505) at org.apache.hadoop.hbase.zookeeper.ZKUtil.createAndFailSilent(ZKUtil.java:829) at org.apache.hadoop.hbase.zookeeper.ZKUtil.createAndFailSilent(ZKUtil.java:819) at org.apache.hadoop.hbase.procedure.ZKProcedureMemberRpcs.sendMemberCompleted(ZKProcedureMemberRpcs.java:276) ... 6 more 2023-04-12 02:55:49,951 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testSplitTableRegion' 2023-04-12 02:55:49,952 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure$1(107): Was KeeperException, not redispatching error org.apache.zookeeper.KeeperException$NoNodeException via jenkins-hbase4.apache.org,43091,1681268031804:org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase4.apache.org,43091,1681268031804 at org.apache.hadoop.hbase.procedure.Subprocedure.cancel(Subprocedure.java:263) at org.apache.hadoop.hbase.procedure.ProcedureMember.controllerConnectionFailure(ProcedureMember.java:214) at org.apache.hadoop.hbase.procedure.ZKProcedureMemberRpcs.sendMemberCompleted(ZKProcedureMemberRpcs.java:279) at org.apache.hadoop.hbase.procedure.Subprocedure.call(Subprocedure.java:186) at org.apache.hadoop.hbase.procedure.Subprocedure.call(Subprocedure.java:46) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase4.apache.org,43091,1681268031804 at org.apache.zookeeper.KeeperException.create(KeeperException.java:118) at org.apache.zookeeper.KeeperException.create(KeeperException.java:54) at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:1538) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.createNonSequential(RecoverableZooKeeper.java:525) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.create(RecoverableZooKeeper.java:505) at org.apache.hadoop.hbase.zookeeper.ZKUtil.createAndFailSilent(ZKUtil.java:829) at org.apache.hadoop.hbase.zookeeper.ZKUtil.createAndFailSilent(ZKUtil.java:819) at org.apache.hadoop.hbase.procedure.ZKProcedureMemberRpcs.sendMemberCompleted(ZKProcedureMemberRpcs.java:276) ... 6 more 2023-04-12 02:55:49,952 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegion' has notified controller of completion 2023-04-12 02:55:49,952 ERROR [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(258): Subprocedure 'testSplitTableRegion' aborting due to a ForeignException! org.apache.zookeeper.KeeperException$NoNodeException via jenkins-hbase4.apache.org,43091,1681268031804:org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase4.apache.org,43091,1681268031804 at org.apache.hadoop.hbase.errorhandling.ForeignExceptionDispatcher.rethrowException(ForeignExceptionDispatcher.java:81) at org.apache.hadoop.hbase.procedure.Subprocedure.rethrowException(Subprocedure.java:137) at org.apache.hadoop.hbase.procedure.Subprocedure.call(Subprocedure.java:190) at org.apache.hadoop.hbase.procedure.Subprocedure.call(Subprocedure.java:46) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase4.apache.org,43091,1681268031804 at org.apache.zookeeper.KeeperException.create(KeeperException.java:118) at org.apache.zookeeper.KeeperException.create(KeeperException.java:54) at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:1538) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.createNonSequential(RecoverableZooKeeper.java:525) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.create(RecoverableZooKeeper.java:505) at org.apache.hadoop.hbase.zookeeper.ZKUtil.createAndFailSilent(ZKUtil.java:829) at org.apache.hadoop.hbase.zookeeper.ZKUtil.createAndFailSilent(ZKUtil.java:819) at org.apache.hadoop.hbase.procedure.ZKProcedureMemberRpcs.sendMemberCompleted(ZKProcedureMemberRpcs.java:276) at org.apache.hadoop.hbase.procedure.Subprocedure.call(Subprocedure.java:186) ... 5 more 2023-04-12 02:55:49,952 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(204): Subprocedure 'testSplitTableRegion' running cleanup. 2023-04-12 02:55:49,952 INFO [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] flush.FlushTableSubprocedure(140): Aborting all flush region subprocedure task threads for 'testSplitTableRegion' due to error org.apache.zookeeper.KeeperException$NoNodeException via jenkins-hbase4.apache.org,43091,1681268031804:org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase4.apache.org,43091,1681268031804 at org.apache.hadoop.hbase.errorhandling.ForeignExceptionDispatcher.rethrowException(ForeignExceptionDispatcher.java:81) at org.apache.hadoop.hbase.procedure.Subprocedure.rethrowException(Subprocedure.java:137) at org.apache.hadoop.hbase.procedure.Subprocedure.call(Subprocedure.java:190) at org.apache.hadoop.hbase.procedure.Subprocedure.call(Subprocedure.java:46) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase4.apache.org,43091,1681268031804 at org.apache.zookeeper.KeeperException.create(KeeperException.java:118) at org.apache.zookeeper.KeeperException.create(KeeperException.java:54) at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:1538) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.createNonSequential(RecoverableZooKeeper.java:525) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.create(RecoverableZooKeeper.java:505) at org.apache.hadoop.hbase.zookeeper.ZKUtil.createAndFailSilent(ZKUtil.java:829) at org.apache.hadoop.hbase.zookeeper.ZKUtil.createAndFailSilent(ZKUtil.java:819) at org.apache.hadoop.hbase.procedure.ZKProcedureMemberRpcs.sendMemberCompleted(ZKProcedureMemberRpcs.java:276) at org.apache.hadoop.hbase.procedure.Subprocedure.call(Subprocedure.java:186) ... 5 more 2023-04-12 02:55:49,953 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:49,953 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:55:49,953 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegion' completed. 2023-04-12 02:55:49,950 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:55:49,953 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:55:49,953 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:55:49,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:55:49,950 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:55:49,950 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:55:49,953 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:55:49,950 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:49,953 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegion 2023-04-12 02:55:49,953 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:55:49,952 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 02:55:49,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:55:49,953 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:49,954 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:55:49,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:55:49,954 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:49,954 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegion 2023-04-12 02:55:49,954 DEBUG [Listener at localhost/46667] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testSplitTableRegion'' to complete. (max 20000 ms per retry) 2023-04-12 02:55:49,954 DEBUG [Listener at localhost/46667] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 02:55:49,954 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:55:49,954 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:49,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:55:49,954 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-12 02:55:49,954 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:49,954 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-12 02:55:49,954 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-12 02:55:50,054 DEBUG [Listener at localhost/46667] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 02:55:50,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 02:55:50,069 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 02:55:50,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] procedure.ProcedureCoordinator(143): Procedure testSplitTableRegion was in running list but was completed. Accepting new attempt. 2023-04-12 02:55:50,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] procedure.ProcedureCoordinator(165): Submitting procedure testSplitTableRegion 2023-04-12 02:55:50,070 INFO [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testSplitTableRegion' 2023-04-12 02:55:50,070 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:55:50,070 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testSplitTableRegion' starting 'acquire' 2023-04-12 02:55:50,070 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testSplitTableRegion', kicking off acquire phase on members. 2023-04-12 02:55:50,071 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-12 02:55:50,071 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testSplitTableRegion 2023-04-12 02:55:50,073 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:50,073 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:55:50,073 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:55:50,073 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:55:50,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:55:50,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:55:50,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:55:50,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:55:50,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:55:50,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:55:50,074 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:50,074 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 02:55:50,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegion 2023-04-12 02:55:50,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegion 2023-04-12 02:55:50,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegion 2023-04-12 02:55:50,074 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-12 02:55:50,074 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-12 02:55:50,075 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-12 02:55:50,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 02:55:50,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegion 2023-04-12 02:55:50,075 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegion 2023-04-12 02:55:50,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 02:55:50,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegion 2023-04-12 02:55:50,075 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegion 2023-04-12 02:55:50,075 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegion is still present, removing 2023-04-12 02:55:50,075 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegion 2023-04-12 02:55:50,075 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegion' with timeout 60000ms 2023-04-12 02:55:50,075 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:55:50,075 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegion' starting 'acquire' stage 2023-04-12 02:55:50,075 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegion is still present, removing 2023-04-12 02:55:50,075 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegion 2023-04-12 02:55:50,076 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegion' with timeout 60000ms 2023-04-12 02:55:50,076 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:55:50,076 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegion' starting 'acquire' stage 2023-04-12 02:55:50,076 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegion' locally acquired 2023-04-12 02:55:50,076 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43091,1681268031804' joining acquired barrier for procedure (testSplitTableRegion) in zk 2023-04-12 02:55:50,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 02:55:50,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegion 2023-04-12 02:55:50,076 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegion 2023-04-12 02:55:50,078 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-12 02:55:50,078 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-12 02:55:50,078 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegion' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 02:55:50,078 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 02:55:50,078 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 02:55:50,078 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegion is still present, removing 2023-04-12 02:55:50,078 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegion 2023-04-12 02:55:50,078 DEBUG [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testSplitTableRegion,,1681268148351.c426152f6c5cdf7d09ab747442f08ea4. 2023-04-12 02:55:50,078 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegion' with timeout 60000ms 2023-04-12 02:55:50,078 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:55:50,078 DEBUG [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testSplitTableRegion,,1681268148351.c426152f6c5cdf7d09ab747442f08ea4. started... 2023-04-12 02:55:50,078 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegion' starting 'acquire' stage 2023-04-12 02:55:50,079 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegion' locally acquired 2023-04-12 02:55:50,079 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44679,1681268031772' joining acquired barrier for procedure (testSplitTableRegion) in zk 2023-04-12 02:55:50,079 INFO [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c426152f6c5cdf7d09ab747442f08ea4 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-04-12 02:55:50,080 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-12 02:55:50,080 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-12 02:55:50,080 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegion' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 02:55:50,094 INFO [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=58 (bloomFilter=true), to=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/.tmp/cf1/a54d3a0a458d49dbbc18bbd89c01e473 2023-04-12 02:55:50,111 INFO [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=58 (bloomFilter=true), to=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/.tmp/cf2/e7cb094353e04c1c89a6b020574e3da1 2023-04-12 02:55:50,116 DEBUG [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/.tmp/cf1/a54d3a0a458d49dbbc18bbd89c01e473 as hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/cf1/a54d3a0a458d49dbbc18bbd89c01e473 2023-04-12 02:55:50,119 INFO [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/cf1/a54d3a0a458d49dbbc18bbd89c01e473, entries=10, sequenceid=58, filesize=5.0 K 2023-04-12 02:55:50,120 DEBUG [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/.tmp/cf2/e7cb094353e04c1c89a6b020574e3da1 as hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/cf2/e7cb094353e04c1c89a6b020574e3da1 2023-04-12 02:55:50,125 INFO [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/cf2/e7cb094353e04c1c89a6b020574e3da1, entries=10, sequenceid=58, filesize=5.0 K 2023-04-12 02:55:50,125 INFO [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for c426152f6c5cdf7d09ab747442f08ea4 in 46ms, sequenceid=58, compaction requested=true 2023-04-12 02:55:50,126 DEBUG [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c426152f6c5cdf7d09ab747442f08ea4: 2023-04-12 02:55:50,126 DEBUG [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testSplitTableRegion,,1681268148351.c426152f6c5cdf7d09ab747442f08ea4. 2023-04-12 02:55:50,126 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 02:55:50,126 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 02:55:50,126 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:50,126 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegion' locally acquired 2023-04-12 02:55:50,126 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34579,1681268031719' joining acquired barrier for procedure (testSplitTableRegion) in zk 2023-04-12 02:55:50,128 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:50,128 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-12 02:55:50,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:50,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 02:55:50,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 02:55:50,128 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-12 02:55:50,128 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegion' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 02:55:50,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 02:55:50,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 02:55:50,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegion 2023-04-12 02:55:50,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:50,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:50,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:50,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 02:55:50,131 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34579,1681268031719' joining acquired barrier for procedure 'testSplitTableRegion' on coordinator 2023-04-12 02:55:50,131 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@40c19e3f[Count = 0] remaining members to acquire global barrier 2023-04-12 02:55:50,131 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testSplitTableRegion' starting 'in-barrier' execution. 2023-04-12 02:55:50,132 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-12 02:55:50,134 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-12 02:55:50,134 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-12 02:55:50,134 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-12 02:55:50,134 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-12 02:55:50,134 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-12 02:55:50,134 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-12 02:55:50,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-12 02:55:50,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-12 02:55:50,134 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegion' received 'reached' from coordinator. 2023-04-12 02:55:50,134 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegion' received 'reached' from coordinator. 2023-04-12 02:55:50,134 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegion' locally completed 2023-04-12 02:55:50,134 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegion' completed for member 'jenkins-hbase4.apache.org,34579,1681268031719' in zk 2023-04-12 02:55:50,134 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegion' locally completed 2023-04-12 02:55:50,134 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:50,134 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 02:55:50,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-12 02:55:50,134 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegion' completed for member 'jenkins-hbase4.apache.org,44679,1681268031772' in zk 2023-04-12 02:55:50,134 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegion' received 'reached' from coordinator. 2023-04-12 02:55:50,135 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegion' locally completed 2023-04-12 02:55:50,135 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegion' completed for member 'jenkins-hbase4.apache.org,43091,1681268031804' in zk 2023-04-12 02:55:50,136 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegion' has notified controller of completion 2023-04-12 02:55:50,136 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:55:50,136 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:50,136 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegion' completed. 2023-04-12 02:55:50,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:50,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 02:55:50,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 02:55:50,137 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegion' has notified controller of completion 2023-04-12 02:55:50,137 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:55:50,137 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegion' has notified controller of completion 2023-04-12 02:55:50,137 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegion' completed. 2023-04-12 02:55:50,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 02:55:50,137 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:55:50,137 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegion' completed. 2023-04-12 02:55:50,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 02:55:50,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegion 2023-04-12 02:55:50,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:50,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:50,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:50,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 02:55:50,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegion 2023-04-12 02:55:50,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:50,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:50,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:50,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testSplitTableRegion' member 'jenkins-hbase4.apache.org,34579,1681268031719': 2023-04-12 02:55:50,143 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34579,1681268031719' released barrier for procedure'testSplitTableRegion', counting down latch. Waiting for 0 more 2023-04-12 02:55:50,143 INFO [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testSplitTableRegion' execution completed 2023-04-12 02:55:50,143 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 02:55:50,143 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 02:55:50,143 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testSplitTableRegion 2023-04-12 02:55:50,143 INFO [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testSplitTableRegionincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 02:55:50,145 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-12 02:55:50,145 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-12 02:55:50,145 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-12 02:55:50,145 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-12 02:55:50,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-12 02:55:50,145 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-12 02:55:50,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-12 02:55:50,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 02:55:50,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 02:55:50,145 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:55:50,145 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-12 02:55:50,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-12 02:55:50,146 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:55:50,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:55:50,145 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:55:50,146 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:50,145 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-12 02:55:50,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-12 02:55:50,145 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:55:50,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 02:55:50,147 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:55:50,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegion 2023-04-12 02:55:50,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:55:50,147 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:55:50,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-12 02:55:50,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:55:50,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 02:55:50,147 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:50,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-12 02:55:50,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-12 02:55:50,148 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:50,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegion 2023-04-12 02:55:50,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:50,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:50,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:50,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 02:55:50,151 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:50,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegion 2023-04-12 02:55:50,151 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:50,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:50,151 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:50,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:50,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:50,154 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:55:50,154 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:55:50,154 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:55:50,154 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:50,154 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:55:50,154 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:55:50,154 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:55:50,154 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:55:50,154 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:55:50,154 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegion 2023-04-12 02:55:50,154 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testSplitTableRegion' 2023-04-12 02:55:50,154 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 02:55:50,154 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:55:50,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:55:50,154 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:55:50,154 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:50,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:55:50,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:55:50,154 DEBUG [Listener at localhost/46667] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testSplitTableRegion'' to complete. (max 20000 ms per retry) 2023-04-12 02:55:50,154 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:50,155 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegion 2023-04-12 02:55:50,155 DEBUG [Listener at localhost/46667] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 02:55:50,155 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:55:50,155 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:55:50,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:55:50,155 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:55:50,155 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:50,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:55:50,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:55:50,155 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-12 02:55:50,155 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:50,155 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:50,155 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-12 02:55:50,155 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-12 02:55:50,163 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-12 02:55:50,255 DEBUG [Listener at localhost/46667] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 02:55:50,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 02:55:50,265 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-12 02:55:50,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] procedure.ProcedureCoordinator(143): Procedure testSplitTableRegion was in running list but was completed. Accepting new attempt. 2023-04-12 02:55:50,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] procedure.ProcedureCoordinator(165): Submitting procedure testSplitTableRegion 2023-04-12 02:55:50,267 INFO [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testSplitTableRegion' 2023-04-12 02:55:50,267 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:55:50,267 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testSplitTableRegion' starting 'acquire' 2023-04-12 02:55:50,267 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testSplitTableRegion', kicking off acquire phase on members. 2023-04-12 02:55:50,267 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-12 02:55:50,268 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testSplitTableRegion 2023-04-12 02:55:50,269 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:55:50,269 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:50,269 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:55:50,269 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:55:50,269 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:55:50,269 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:55:50,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:55:50,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:55:50,269 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:55:50,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:55:50,269 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:50,269 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-12 02:55:50,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegion 2023-04-12 02:55:50,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegion 2023-04-12 02:55:50,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegion 2023-04-12 02:55:50,270 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-12 02:55:50,270 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-12 02:55:50,270 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-12 02:55:50,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 02:55:50,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegion 2023-04-12 02:55:50,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 02:55:50,270 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegion 2023-04-12 02:55:50,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegion 2023-04-12 02:55:50,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-12 02:55:50,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegion 2023-04-12 02:55:50,270 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegion 2023-04-12 02:55:50,270 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegion 2023-04-12 02:55:50,271 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegion is still present, removing 2023-04-12 02:55:50,271 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegion 2023-04-12 02:55:50,271 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegion is still present, removing 2023-04-12 02:55:50,271 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegion' with timeout 60000ms 2023-04-12 02:55:50,271 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:55:50,271 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegion 2023-04-12 02:55:50,271 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegion' starting 'acquire' stage 2023-04-12 02:55:50,271 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegion' with timeout 60000ms 2023-04-12 02:55:50,271 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:55:50,271 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegion is still present, removing 2023-04-12 02:55:50,271 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegion' starting 'acquire' stage 2023-04-12 02:55:50,271 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegion 2023-04-12 02:55:50,271 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegion' locally acquired 2023-04-12 02:55:50,271 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,43091,1681268031804' joining acquired barrier for procedure (testSplitTableRegion) in zk 2023-04-12 02:55:50,271 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegion' with timeout 60000ms 2023-04-12 02:55:50,271 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-12 02:55:50,271 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-12 02:55:50,271 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-12 02:55:50,271 DEBUG [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testSplitTableRegion,,1681268148351.c426152f6c5cdf7d09ab747442f08ea4. 2023-04-12 02:55:50,271 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegion' starting 'acquire' stage 2023-04-12 02:55:50,271 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegion' locally acquired 2023-04-12 02:55:50,271 DEBUG [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testSplitTableRegion,,1681268148351.c426152f6c5cdf7d09ab747442f08ea4. started... 2023-04-12 02:55:50,271 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,44679,1681268031772' joining acquired barrier for procedure (testSplitTableRegion) in zk 2023-04-12 02:55:50,272 INFO [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c426152f6c5cdf7d09ab747442f08ea4 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-04-12 02:55:50,272 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-12 02:55:50,273 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-12 02:55:50,273 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegion' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 02:55:50,274 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-12 02:55:50,275 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-12 02:55:50,275 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegion' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 02:55:50,282 INFO [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=71 (bloomFilter=true), to=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/.tmp/cf1/79de355eab9c46f98c089a49fcf5f846 2023-04-12 02:55:50,291 INFO [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=71 (bloomFilter=true), to=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/.tmp/cf2/506165cd013640888b0ebd392bd533ab 2023-04-12 02:55:50,294 DEBUG [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/.tmp/cf1/79de355eab9c46f98c089a49fcf5f846 as hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/cf1/79de355eab9c46f98c089a49fcf5f846 2023-04-12 02:55:50,297 INFO [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/cf1/79de355eab9c46f98c089a49fcf5f846, entries=10, sequenceid=71, filesize=5.0 K 2023-04-12 02:55:50,297 DEBUG [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/.tmp/cf2/506165cd013640888b0ebd392bd533ab as hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/cf2/506165cd013640888b0ebd392bd533ab 2023-04-12 02:55:50,300 INFO [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/cf2/506165cd013640888b0ebd392bd533ab, entries=10, sequenceid=71, filesize=5.0 K 2023-04-12 02:55:50,301 INFO [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for c426152f6c5cdf7d09ab747442f08ea4 in 28ms, sequenceid=71, compaction requested=true 2023-04-12 02:55:50,301 DEBUG [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c426152f6c5cdf7d09ab747442f08ea4: 2023-04-12 02:55:50,301 DEBUG [rs(jenkins-hbase4.apache.org,34579,1681268031719)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testSplitTableRegion,,1681268148351.c426152f6c5cdf7d09ab747442f08ea4. 2023-04-12 02:55:50,301 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-12 02:55:50,301 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-12 02:55:50,301 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:50,301 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegion' locally acquired 2023-04-12 02:55:50,301 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,34579,1681268031719' joining acquired barrier for procedure (testSplitTableRegion) in zk 2023-04-12 02:55:50,302 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:50,302 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-12 02:55:50,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:50,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 02:55:50,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 02:55:50,302 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-12 02:55:50,303 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegion' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-12 02:55:50,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 02:55:50,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 02:55:50,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegion 2023-04-12 02:55:50,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:50,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:50,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:50,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 02:55:50,304 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,34579,1681268031719' joining acquired barrier for procedure 'testSplitTableRegion' on coordinator 2023-04-12 02:55:50,304 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@648d1f81[Count = 0] remaining members to acquire global barrier 2023-04-12 02:55:50,304 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testSplitTableRegion' starting 'in-barrier' execution. 2023-04-12 02:55:50,304 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-12 02:55:50,305 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-12 02:55:50,305 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-12 02:55:50,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-12 02:55:50,305 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-12 02:55:50,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-12 02:55:50,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-12 02:55:50,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-12 02:55:50,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-12 02:55:50,305 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegion' received 'reached' from coordinator. 2023-04-12 02:55:50,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-12 02:55:50,305 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:50,306 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegion' received 'reached' from coordinator. 2023-04-12 02:55:50,306 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegion' locally completed 2023-04-12 02:55:50,306 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegion' completed for member 'jenkins-hbase4.apache.org,43091,1681268031804' in zk 2023-04-12 02:55:50,305 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegion' received 'reached' from coordinator. 2023-04-12 02:55:50,306 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegion' locally completed 2023-04-12 02:55:50,306 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegion' completed for member 'jenkins-hbase4.apache.org,34579,1681268031719' in zk 2023-04-12 02:55:50,306 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-12 02:55:50,305 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegion' locally completed 2023-04-12 02:55:50,306 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegion' completed for member 'jenkins-hbase4.apache.org,44679,1681268031772' in zk 2023-04-12 02:55:50,310 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegion' has notified controller of completion 2023-04-12 02:55:50,310 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegion' has notified controller of completion 2023-04-12 02:55:50,310 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:55:50,310 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:50,310 DEBUG [member: 'jenkins-hbase4.apache.org,34579,1681268031719' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegion' completed. 2023-04-12 02:55:50,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:50,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 02:55:50,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 02:55:50,310 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:55:50,310 DEBUG [member: 'jenkins-hbase4.apache.org,43091,1681268031804' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegion' completed. 2023-04-12 02:55:50,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 02:55:50,310 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegion' has notified controller of completion 2023-04-12 02:55:50,310 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:55:50,310 DEBUG [member: 'jenkins-hbase4.apache.org,44679,1681268031772' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegion' completed. 2023-04-12 02:55:50,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 02:55:50,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegion 2023-04-12 02:55:50,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:50,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:50,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:50,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 02:55:50,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegion 2023-04-12 02:55:50,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:50,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:50,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:50,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testSplitTableRegion' member 'jenkins-hbase4.apache.org,34579,1681268031719': 2023-04-12 02:55:50,313 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,34579,1681268031719' released barrier for procedure'testSplitTableRegion', counting down latch. Waiting for 0 more 2023-04-12 02:55:50,313 INFO [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testSplitTableRegion' execution completed 2023-04-12 02:55:50,313 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-12 02:55:50,313 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-12 02:55:50,313 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testSplitTableRegion 2023-04-12 02:55:50,313 INFO [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testSplitTableRegionincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-12 02:55:50,314 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-12 02:55:50,314 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-12 02:55:50,314 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-12 02:55:50,315 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-12 02:55:50,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-12 02:55:50,315 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-12 02:55:50,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-12 02:55:50,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-12 02:55:50,315 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:55:50,315 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-12 02:55:50,315 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:55:50,314 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-12 02:55:50,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-12 02:55:50,314 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:55:50,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-12 02:55:50,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-12 02:55:50,315 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:55:50,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:55:50,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-12 02:55:50,315 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:50,315 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:55:50,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegion 2023-04-12 02:55:50,316 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:55:50,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:55:50,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-12 02:55:50,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:55:50,316 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:50,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-12 02:55:50,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-12 02:55:50,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-12 02:55:50,316 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:50,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegion 2023-04-12 02:55:50,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:50,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:50,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:50,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-12 02:55:50,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegion 2023-04-12 02:55:50,318 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:50,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:50,318 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:50,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:50,318 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:50,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:50,321 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:55:50,321 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:55:50,321 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-12 02:55:50,321 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:55:50,321 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:55:50,321 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:55:50,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:55:50,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:55:50,321 DEBUG [(jenkins-hbase4.apache.org,40281,1681268030476)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-12 02:55:50,321 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:55:50,321 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-12 02:55:50,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-12 02:55:50,321 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-12 02:55:50,321 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:50,321 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testSplitTableRegion' 2023-04-12 02:55:50,321 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:55:50,321 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:55:50,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:55:50,321 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-12 02:55:50,321 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegion 2023-04-12 02:55:50,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-12 02:55:50,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:55:50,322 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:50,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-12 02:55:50,322 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:50,322 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegion 2023-04-12 02:55:50,322 DEBUG [Listener at localhost/46667] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testSplitTableRegion'' to complete. (max 20000 ms per retry) 2023-04-12 02:55:50,322 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:50,322 DEBUG [Listener at localhost/46667] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-12 02:55:50,322 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-12 02:55:50,322 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:50,322 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:50,322 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-12 02:55:50,322 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-12 02:55:50,422 DEBUG [Listener at localhost/46667] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-12 02:55:50,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-12 02:55:50,430 DEBUG [Listener at localhost/46667] assignment.SplitTableRegionProcedure(219): Splittable=true state=OPEN, location=jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:50,430 DEBUG [Listener at localhost/46667] procedure2.ProcedureExecutor(1029): Stored pid=144, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testSplitTableRegion, parent=c426152f6c5cdf7d09ab747442f08ea4, daughterA=9a391ed3cdb851dca0e4ca65b3a980e8, daughterB=c9ca1dc933715e878706f11781755586 2023-04-12 02:55:50,431 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=144, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testSplitTableRegion, parent=c426152f6c5cdf7d09ab747442f08ea4, daughterA=9a391ed3cdb851dca0e4ca65b3a980e8, daughterB=c9ca1dc933715e878706f11781755586 2023-04-12 02:55:50,431 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=144, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testSplitTableRegion, parent=c426152f6c5cdf7d09ab747442f08ea4, daughterA=9a391ed3cdb851dca0e4ca65b3a980e8, daughterB=c9ca1dc933715e878706f11781755586 2023-04-12 02:55:50,432 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=144, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testSplitTableRegion, parent=c426152f6c5cdf7d09ab747442f08ea4, daughterA=9a391ed3cdb851dca0e4ca65b3a980e8, daughterB=c9ca1dc933715e878706f11781755586 2023-04-12 02:55:50,433 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=145, ppid=144, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitTableRegion, region=c426152f6c5cdf7d09ab747442f08ea4, UNASSIGN}] 2023-04-12 02:55:50,434 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=145, ppid=144, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitTableRegion, region=c426152f6c5cdf7d09ab747442f08ea4, UNASSIGN 2023-04-12 02:55:50,434 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=145 updating hbase:meta row=c426152f6c5cdf7d09ab747442f08ea4, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:50,434 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitTableRegion,,1681268148351.c426152f6c5cdf7d09ab747442f08ea4.","families":{"info":[{"qualifier":"regioninfo","vlen":54,"tag":[],"timestamp":"1681268150434"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681268150434"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681268150434"}]},"ts":"1681268150434"} 2023-04-12 02:55:50,435 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=146, ppid=145, state=RUNNABLE; CloseRegionProcedure c426152f6c5cdf7d09ab747442f08ea4, server=jenkins-hbase4.apache.org,34579,1681268031719}] 2023-04-12 02:55:50,586 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close c426152f6c5cdf7d09ab747442f08ea4 2023-04-12 02:55:50,587 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing c426152f6c5cdf7d09ab747442f08ea4, disabling compactions & flushes 2023-04-12 02:55:50,587 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region testSplitTableRegion,,1681268148351.c426152f6c5cdf7d09ab747442f08ea4. 2023-04-12 02:55:50,587 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testSplitTableRegion,,1681268148351.c426152f6c5cdf7d09ab747442f08ea4. 2023-04-12 02:55:50,587 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on testSplitTableRegion,,1681268148351.c426152f6c5cdf7d09ab747442f08ea4. after waiting 0 ms 2023-04-12 02:55:50,587 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region testSplitTableRegion,,1681268148351.c426152f6c5cdf7d09ab747442f08ea4. 2023-04-12 02:55:50,587 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing c426152f6c5cdf7d09ab747442f08ea4 2/2 column families, dataSize=480 B heapSize=2.13 KB 2023-04-12 02:55:50,995 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=240 B at sequenceid=82 (bloomFilter=true), to=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/.tmp/cf1/1f290c1ae6ff4ac9aa79371af93b04cf 2023-04-12 02:55:51,005 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=240 B at sequenceid=82 (bloomFilter=true), to=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/.tmp/cf2/5baff79a42ff479888e579a42af26748 2023-04-12 02:55:51,009 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/.tmp/cf1/1f290c1ae6ff4ac9aa79371af93b04cf as hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/cf1/1f290c1ae6ff4ac9aa79371af93b04cf 2023-04-12 02:55:51,012 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/cf1/1f290c1ae6ff4ac9aa79371af93b04cf, entries=8, sequenceid=82, filesize=5.0 K 2023-04-12 02:55:51,012 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/.tmp/cf2/5baff79a42ff479888e579a42af26748 as hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/cf2/5baff79a42ff479888e579a42af26748 2023-04-12 02:55:51,015 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/cf2/5baff79a42ff479888e579a42af26748, entries=8, sequenceid=82, filesize=5.0 K 2023-04-12 02:55:51,016 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~480 B/480, heapSize ~2.09 KB/2144, currentSize=0 B/0 for c426152f6c5cdf7d09ab747442f08ea4 in 429ms, sequenceid=82, compaction requested=true 2023-04-12 02:55:51,019 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/recovered.edits/85.seqid, newMaxSeqId=85, maxSeqId=1 2023-04-12 02:55:51,020 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro 2023-04-12 02:55:51,021 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed testSplitTableRegion,,1681268148351.c426152f6c5cdf7d09ab747442f08ea4. 2023-04-12 02:55:51,021 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for c426152f6c5cdf7d09ab747442f08ea4: 2023-04-12 02:55:51,022 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed c426152f6c5cdf7d09ab747442f08ea4 2023-04-12 02:55:51,022 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=145 updating hbase:meta row=c426152f6c5cdf7d09ab747442f08ea4, regionState=CLOSED 2023-04-12 02:55:51,023 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testSplitTableRegion,,1681268148351.c426152f6c5cdf7d09ab747442f08ea4.","families":{"info":[{"qualifier":"regioninfo","vlen":54,"tag":[],"timestamp":"1681268151022"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681268151022"}]},"ts":"1681268151022"} 2023-04-12 02:55:51,025 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=146, resume processing ppid=145 2023-04-12 02:55:51,025 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=146, ppid=145, state=SUCCESS; CloseRegionProcedure c426152f6c5cdf7d09ab747442f08ea4, server=jenkins-hbase4.apache.org,34579,1681268031719 in 589 msec 2023-04-12 02:55:51,026 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=145, resume processing ppid=144 2023-04-12 02:55:51,026 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=145, ppid=144, state=SUCCESS; TransitRegionStateProcedure table=testSplitTableRegion, region=c426152f6c5cdf7d09ab747442f08ea4, UNASSIGN in 592 msec 2023-04-12 02:55:51,034 INFO [PEWorker-1] assignment.SplitTableRegionProcedure(694): pid=144 splitting 14 storefiles, region=c426152f6c5cdf7d09ab747442f08ea4, threads=14 2023-04-12 02:55:51,034 DEBUG [StoreFileSplitter-pool-0] assignment.SplitTableRegionProcedure(776): pid=144 splitting started for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/cf2/506165cd013640888b0ebd392bd533ab for region: c426152f6c5cdf7d09ab747442f08ea4 2023-04-12 02:55:51,035 DEBUG [StoreFileSplitter-pool-1] assignment.SplitTableRegionProcedure(776): pid=144 splitting started for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/cf2/5baff79a42ff479888e579a42af26748 for region: c426152f6c5cdf7d09ab747442f08ea4 2023-04-12 02:55:51,035 DEBUG [StoreFileSplitter-pool-2] assignment.SplitTableRegionProcedure(776): pid=144 splitting started for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/cf2/ba426c6436064f9ca5ca16cecc0d8c98 for region: c426152f6c5cdf7d09ab747442f08ea4 2023-04-12 02:55:51,035 DEBUG [StoreFileSplitter-pool-3] assignment.SplitTableRegionProcedure(776): pid=144 splitting started for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/cf2/c4a22b15c4714bc1a6f7b206d3315186 for region: c426152f6c5cdf7d09ab747442f08ea4 2023-04-12 02:55:51,035 DEBUG [StoreFileSplitter-pool-4] assignment.SplitTableRegionProcedure(776): pid=144 splitting started for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/cf2/e4c7205c2e024488b0db4154888f3f50 for region: c426152f6c5cdf7d09ab747442f08ea4 2023-04-12 02:55:51,035 DEBUG [StoreFileSplitter-pool-5] assignment.SplitTableRegionProcedure(776): pid=144 splitting started for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/cf2/e7cb094353e04c1c89a6b020574e3da1 for region: c426152f6c5cdf7d09ab747442f08ea4 2023-04-12 02:55:51,035 DEBUG [StoreFileSplitter-pool-7] assignment.SplitTableRegionProcedure(776): pid=144 splitting started for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/cf1/1f290c1ae6ff4ac9aa79371af93b04cf for region: c426152f6c5cdf7d09ab747442f08ea4 2023-04-12 02:55:51,035 DEBUG [StoreFileSplitter-pool-6] assignment.SplitTableRegionProcedure(776): pid=144 splitting started for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/cf2/f5ea9d0c0a4c491f91c40210e3a13892 for region: c426152f6c5cdf7d09ab747442f08ea4 2023-04-12 02:55:51,035 DEBUG [StoreFileSplitter-pool-8] assignment.SplitTableRegionProcedure(776): pid=144 splitting started for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/cf1/3e014fee55b841d9a93f29d49aa55156 for region: c426152f6c5cdf7d09ab747442f08ea4 2023-04-12 02:55:51,035 DEBUG [StoreFileSplitter-pool-9] assignment.SplitTableRegionProcedure(776): pid=144 splitting started for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/cf1/52c29c1aa57f4576b791c97686dd2e0d for region: c426152f6c5cdf7d09ab747442f08ea4 2023-04-12 02:55:51,035 DEBUG [StoreFileSplitter-pool-10] assignment.SplitTableRegionProcedure(776): pid=144 splitting started for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/cf1/72636f8bfb49470784b49ea8615a5c6a for region: c426152f6c5cdf7d09ab747442f08ea4 2023-04-12 02:55:51,036 DEBUG [StoreFileSplitter-pool-11] assignment.SplitTableRegionProcedure(776): pid=144 splitting started for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/cf1/79de355eab9c46f98c089a49fcf5f846 for region: c426152f6c5cdf7d09ab747442f08ea4 2023-04-12 02:55:51,036 DEBUG [StoreFileSplitter-pool-12] assignment.SplitTableRegionProcedure(776): pid=144 splitting started for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/cf1/a54d3a0a458d49dbbc18bbd89c01e473 for region: c426152f6c5cdf7d09ab747442f08ea4 2023-04-12 02:55:51,036 DEBUG [StoreFileSplitter-pool-13] assignment.SplitTableRegionProcedure(776): pid=144 splitting started for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/cf1/bd8f39350eae4e18bebd947afb5d9f5e for region: c426152f6c5cdf7d09ab747442f08ea4 2023-04-12 02:55:51,172 DEBUG [StoreFileSplitter-pool-2] assignment.SplitTableRegionProcedure(787): pid=144 splitting complete for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/cf2/ba426c6436064f9ca5ca16cecc0d8c98 for region: c426152f6c5cdf7d09ab747442f08ea4 2023-04-12 02:55:51,192 DEBUG [StoreFileSplitter-pool-7] assignment.SplitTableRegionProcedure(787): pid=144 splitting complete for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/cf1/1f290c1ae6ff4ac9aa79371af93b04cf for region: c426152f6c5cdf7d09ab747442f08ea4 2023-04-12 02:55:51,200 DEBUG [StoreFileSplitter-pool-13] assignment.SplitTableRegionProcedure(787): pid=144 splitting complete for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/cf1/bd8f39350eae4e18bebd947afb5d9f5e for region: c426152f6c5cdf7d09ab747442f08ea4 2023-04-12 02:55:51,206 DEBUG [StoreFileSplitter-pool-9] assignment.SplitTableRegionProcedure(787): pid=144 splitting complete for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/cf1/52c29c1aa57f4576b791c97686dd2e0d for region: c426152f6c5cdf7d09ab747442f08ea4 2023-04-12 02:55:51,212 DEBUG [StoreFileSplitter-pool-1] assignment.SplitTableRegionProcedure(787): pid=144 splitting complete for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/cf2/5baff79a42ff479888e579a42af26748 for region: c426152f6c5cdf7d09ab747442f08ea4 2023-04-12 02:55:51,224 DEBUG [StoreFileSplitter-pool-4] assignment.SplitTableRegionProcedure(787): pid=144 splitting complete for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/cf2/e4c7205c2e024488b0db4154888f3f50 for region: c426152f6c5cdf7d09ab747442f08ea4 2023-04-12 02:55:51,232 DEBUG [StoreFileSplitter-pool-6] assignment.SplitTableRegionProcedure(787): pid=144 splitting complete for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/cf2/f5ea9d0c0a4c491f91c40210e3a13892 for region: c426152f6c5cdf7d09ab747442f08ea4 2023-04-12 02:55:51,232 DEBUG [StoreFileSplitter-pool-5] assignment.SplitTableRegionProcedure(787): pid=144 splitting complete for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/cf2/e7cb094353e04c1c89a6b020574e3da1 for region: c426152f6c5cdf7d09ab747442f08ea4 2023-04-12 02:55:51,232 DEBUG [StoreFileSplitter-pool-8] assignment.SplitTableRegionProcedure(787): pid=144 splitting complete for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/cf1/3e014fee55b841d9a93f29d49aa55156 for region: c426152f6c5cdf7d09ab747442f08ea4 2023-04-12 02:55:51,234 DEBUG [StoreFileSplitter-pool-12] assignment.SplitTableRegionProcedure(787): pid=144 splitting complete for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/cf1/a54d3a0a458d49dbbc18bbd89c01e473 for region: c426152f6c5cdf7d09ab747442f08ea4 2023-04-12 02:55:51,548 DEBUG [StoreFileSplitter-pool-3] assignment.SplitTableRegionProcedure(787): pid=144 splitting complete for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/cf2/c4a22b15c4714bc1a6f7b206d3315186 for region: c426152f6c5cdf7d09ab747442f08ea4 2023-04-12 02:55:51,548 DEBUG [StoreFileSplitter-pool-11] assignment.SplitTableRegionProcedure(787): pid=144 splitting complete for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/cf1/79de355eab9c46f98c089a49fcf5f846 for region: c426152f6c5cdf7d09ab747442f08ea4 2023-04-12 02:55:51,548 DEBUG [StoreFileSplitter-pool-0] assignment.SplitTableRegionProcedure(787): pid=144 splitting complete for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/cf2/506165cd013640888b0ebd392bd533ab for region: c426152f6c5cdf7d09ab747442f08ea4 2023-04-12 02:55:51,622 DEBUG [StoreFileSplitter-pool-10] assignment.SplitTableRegionProcedure(787): pid=144 splitting complete for store file: hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/cf1/72636f8bfb49470784b49ea8615a5c6a for region: c426152f6c5cdf7d09ab747442f08ea4 2023-04-12 02:55:51,622 DEBUG [PEWorker-1] assignment.SplitTableRegionProcedure(755): pid=144 split storefiles for region c426152f6c5cdf7d09ab747442f08ea4 Daughter A: 14 storefiles, Daughter B: 14 storefiles. 2023-04-12 02:55:51,653 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/9a391ed3cdb851dca0e4ca65b3a980e8/recovered.edits/85.seqid, newMaxSeqId=85, maxSeqId=-1 2023-04-12 02:55:51,655 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c9ca1dc933715e878706f11781755586/recovered.edits/85.seqid, newMaxSeqId=85, maxSeqId=-1 2023-04-12 02:55:51,656 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":3,"row":"testSplitTableRegion,,1681268148351.c426152f6c5cdf7d09ab747442f08ea4.","families":{"info":[{"qualifier":"regioninfo","vlen":54,"tag":[],"timestamp":"1681268151656"},{"qualifier":"splitA","vlen":56,"tag":[],"timestamp":"1681268151656"},{"qualifier":"splitB","vlen":56,"tag":[],"timestamp":"1681268151656"}]},"ts":"1681268151656"} 2023-04-12 02:55:51,656 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":3,"row":"testSplitTableRegion,,1681268150430.9a391ed3cdb851dca0e4ca65b3a980e8.","families":{"info":[{"qualifier":"regioninfo","vlen":56,"tag":[],"timestamp":"1681268151656"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681268151656"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681268151656"}]},"ts":"1681268151656"} 2023-04-12 02:55:51,657 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":3,"row":"testSplitTableRegion,41,1681268150430.c9ca1dc933715e878706f11781755586.","families":{"info":[{"qualifier":"regioninfo","vlen":56,"tag":[],"timestamp":"1681268151656"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681268151656"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681268151656"}]},"ts":"1681268151656"} 2023-04-12 02:55:51,659 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=147, ppid=144, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSplitTableRegion, region=9a391ed3cdb851dca0e4ca65b3a980e8, ASSIGN}, {pid=148, ppid=144, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSplitTableRegion, region=c9ca1dc933715e878706f11781755586, ASSIGN}] 2023-04-12 02:55:51,660 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=148, ppid=144, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSplitTableRegion, region=c9ca1dc933715e878706f11781755586, ASSIGN 2023-04-12 02:55:51,661 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=148, ppid=144, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testSplitTableRegion, region=c9ca1dc933715e878706f11781755586, ASSIGN; state=SPLITTING_NEW, location=jenkins-hbase4.apache.org,34579,1681268031719; forceNewPlan=false, retain=false 2023-04-12 02:55:51,661 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=147, ppid=144, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSplitTableRegion, region=9a391ed3cdb851dca0e4ca65b3a980e8, ASSIGN 2023-04-12 02:55:51,662 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=147, ppid=144, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testSplitTableRegion, region=9a391ed3cdb851dca0e4ca65b3a980e8, ASSIGN; state=SPLITTING_NEW, location=jenkins-hbase4.apache.org,34579,1681268031719; forceNewPlan=false, retain=false 2023-04-12 02:55:51,811 INFO [jenkins-hbase4:40281] balancer.BaseLoadBalancer(1545): Reassigned 2 regions. 2 retained the pre-restart assignment. 2023-04-12 02:55:51,814 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=148 updating hbase:meta row=c9ca1dc933715e878706f11781755586, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:51,814 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitTableRegion,41,1681268150430.c9ca1dc933715e878706f11781755586.","families":{"info":[{"qualifier":"regioninfo","vlen":56,"tag":[],"timestamp":"1681268151813"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681268151813"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681268151813"}]},"ts":"1681268151813"} 2023-04-12 02:55:51,815 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=149, ppid=148, state=RUNNABLE; OpenRegionProcedure c9ca1dc933715e878706f11781755586, server=jenkins-hbase4.apache.org,34579,1681268031719}] 2023-04-12 02:55:51,817 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=147 updating hbase:meta row=9a391ed3cdb851dca0e4ca65b3a980e8, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:51,817 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitTableRegion,,1681268150430.9a391ed3cdb851dca0e4ca65b3a980e8.","families":{"info":[{"qualifier":"regioninfo","vlen":56,"tag":[],"timestamp":"1681268151817"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681268151817"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681268151817"}]},"ts":"1681268151817"} 2023-04-12 02:55:51,818 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=150, ppid=147, state=RUNNABLE; OpenRegionProcedure 9a391ed3cdb851dca0e4ca65b3a980e8, server=jenkins-hbase4.apache.org,34579,1681268031719}] 2023-04-12 02:55:51,969 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testSplitTableRegion,41,1681268150430.c9ca1dc933715e878706f11781755586. 2023-04-12 02:55:51,969 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => c9ca1dc933715e878706f11781755586, NAME => 'testSplitTableRegion,41,1681268150430.c9ca1dc933715e878706f11781755586.', STARTKEY => '41', ENDKEY => ''} 2023-04-12 02:55:51,969 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(174): System coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro loaded, priority=536870911. 2023-04-12 02:55:51,969 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testSplitTableRegion c9ca1dc933715e878706f11781755586 2023-04-12 02:55:51,969 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated testSplitTableRegion,41,1681268150430.c9ca1dc933715e878706f11781755586.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-12 02:55:51,969 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for c9ca1dc933715e878706f11781755586 2023-04-12 02:55:51,969 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for c9ca1dc933715e878706f11781755586 2023-04-12 02:55:51,970 INFO [StoreOpener-c9ca1dc933715e878706f11781755586-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf1 of region c9ca1dc933715e878706f11781755586 2023-04-12 02:55:51,971 DEBUG [StoreOpener-c9ca1dc933715e878706f11781755586-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c9ca1dc933715e878706f11781755586/cf1 2023-04-12 02:55:51,971 DEBUG [StoreOpener-c9ca1dc933715e878706f11781755586-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c9ca1dc933715e878706f11781755586/cf1 2023-04-12 02:55:51,971 INFO [StoreOpener-c9ca1dc933715e878706f11781755586-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region c9ca1dc933715e878706f11781755586 columnFamilyName cf1 2023-04-12 02:55:51,982 DEBUG [StoreOpener-c9ca1dc933715e878706f11781755586-1] regionserver.HStore(539): loaded hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c9ca1dc933715e878706f11781755586/cf1/1f290c1ae6ff4ac9aa79371af93b04cf.c426152f6c5cdf7d09ab747442f08ea4->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/cf1/1f290c1ae6ff4ac9aa79371af93b04cf-top 2023-04-12 02:55:51,984 DEBUG [StoreOpener-c9ca1dc933715e878706f11781755586-1] regionserver.HStore(539): loaded hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c9ca1dc933715e878706f11781755586/cf1/3e014fee55b841d9a93f29d49aa55156.c426152f6c5cdf7d09ab747442f08ea4->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/cf1/3e014fee55b841d9a93f29d49aa55156-top 2023-04-12 02:55:51,987 DEBUG [StoreOpener-c9ca1dc933715e878706f11781755586-1] regionserver.HStore(539): loaded hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c9ca1dc933715e878706f11781755586/cf1/52c29c1aa57f4576b791c97686dd2e0d.c426152f6c5cdf7d09ab747442f08ea4->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/cf1/52c29c1aa57f4576b791c97686dd2e0d-top 2023-04-12 02:55:51,990 DEBUG [StoreOpener-c9ca1dc933715e878706f11781755586-1] regionserver.HStore(539): loaded hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c9ca1dc933715e878706f11781755586/cf1/72636f8bfb49470784b49ea8615a5c6a.c426152f6c5cdf7d09ab747442f08ea4->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/cf1/72636f8bfb49470784b49ea8615a5c6a-top 2023-04-12 02:55:51,993 DEBUG [StoreOpener-c9ca1dc933715e878706f11781755586-1] regionserver.HStore(539): loaded hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c9ca1dc933715e878706f11781755586/cf1/79de355eab9c46f98c089a49fcf5f846.c426152f6c5cdf7d09ab747442f08ea4->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/cf1/79de355eab9c46f98c089a49fcf5f846-top 2023-04-12 02:55:51,995 DEBUG [StoreOpener-c9ca1dc933715e878706f11781755586-1] regionserver.HStore(539): loaded hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c9ca1dc933715e878706f11781755586/cf1/a54d3a0a458d49dbbc18bbd89c01e473.c426152f6c5cdf7d09ab747442f08ea4->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/cf1/a54d3a0a458d49dbbc18bbd89c01e473-top 2023-04-12 02:55:51,998 DEBUG [StoreOpener-c9ca1dc933715e878706f11781755586-1] regionserver.HStore(539): loaded hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c9ca1dc933715e878706f11781755586/cf1/bd8f39350eae4e18bebd947afb5d9f5e.c426152f6c5cdf7d09ab747442f08ea4->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/cf1/bd8f39350eae4e18bebd947afb5d9f5e-top 2023-04-12 02:55:51,998 INFO [StoreOpener-c9ca1dc933715e878706f11781755586-1] regionserver.HStore(310): Store=c9ca1dc933715e878706f11781755586/cf1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-12 02:55:51,998 INFO [StoreOpener-c9ca1dc933715e878706f11781755586-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf2 of region c9ca1dc933715e878706f11781755586 2023-04-12 02:55:51,999 DEBUG [StoreOpener-c9ca1dc933715e878706f11781755586-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c9ca1dc933715e878706f11781755586/cf2 2023-04-12 02:55:51,999 DEBUG [StoreOpener-c9ca1dc933715e878706f11781755586-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c9ca1dc933715e878706f11781755586/cf2 2023-04-12 02:55:52,000 INFO [StoreOpener-c9ca1dc933715e878706f11781755586-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region c9ca1dc933715e878706f11781755586 columnFamilyName cf2 2023-04-12 02:55:52,010 DEBUG [StoreOpener-c9ca1dc933715e878706f11781755586-1] regionserver.HStore(539): loaded hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c9ca1dc933715e878706f11781755586/cf2/506165cd013640888b0ebd392bd533ab.c426152f6c5cdf7d09ab747442f08ea4->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/cf2/506165cd013640888b0ebd392bd533ab-top 2023-04-12 02:55:52,012 DEBUG [StoreOpener-c9ca1dc933715e878706f11781755586-1] regionserver.HStore(539): loaded hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c9ca1dc933715e878706f11781755586/cf2/5baff79a42ff479888e579a42af26748.c426152f6c5cdf7d09ab747442f08ea4->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/cf2/5baff79a42ff479888e579a42af26748-top 2023-04-12 02:55:52,015 DEBUG [StoreOpener-c9ca1dc933715e878706f11781755586-1] regionserver.HStore(539): loaded hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c9ca1dc933715e878706f11781755586/cf2/ba426c6436064f9ca5ca16cecc0d8c98.c426152f6c5cdf7d09ab747442f08ea4->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/cf2/ba426c6436064f9ca5ca16cecc0d8c98-top 2023-04-12 02:55:52,018 DEBUG [StoreOpener-c9ca1dc933715e878706f11781755586-1] regionserver.HStore(539): loaded hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c9ca1dc933715e878706f11781755586/cf2/c4a22b15c4714bc1a6f7b206d3315186.c426152f6c5cdf7d09ab747442f08ea4->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/cf2/c4a22b15c4714bc1a6f7b206d3315186-top 2023-04-12 02:55:52,021 DEBUG [StoreOpener-c9ca1dc933715e878706f11781755586-1] regionserver.HStore(539): loaded hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c9ca1dc933715e878706f11781755586/cf2/e4c7205c2e024488b0db4154888f3f50.c426152f6c5cdf7d09ab747442f08ea4->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/cf2/e4c7205c2e024488b0db4154888f3f50-top 2023-04-12 02:55:52,023 DEBUG [StoreOpener-c9ca1dc933715e878706f11781755586-1] regionserver.HStore(539): loaded hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c9ca1dc933715e878706f11781755586/cf2/e7cb094353e04c1c89a6b020574e3da1.c426152f6c5cdf7d09ab747442f08ea4->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/cf2/e7cb094353e04c1c89a6b020574e3da1-top 2023-04-12 02:55:52,026 DEBUG [StoreOpener-c9ca1dc933715e878706f11781755586-1] regionserver.HStore(539): loaded hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c9ca1dc933715e878706f11781755586/cf2/f5ea9d0c0a4c491f91c40210e3a13892.c426152f6c5cdf7d09ab747442f08ea4->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/cf2/f5ea9d0c0a4c491f91c40210e3a13892-top 2023-04-12 02:55:52,026 INFO [StoreOpener-c9ca1dc933715e878706f11781755586-1] regionserver.HStore(310): Store=c9ca1dc933715e878706f11781755586/cf2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-12 02:55:52,027 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c9ca1dc933715e878706f11781755586 2023-04-12 02:55:52,028 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c9ca1dc933715e878706f11781755586 2023-04-12 02:55:52,029 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testSplitTableRegion descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-04-12 02:55:52,030 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for c9ca1dc933715e878706f11781755586 2023-04-12 02:55:52,031 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened c9ca1dc933715e878706f11781755586; next sequenceid=86; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=59092989, jitterRate=-0.11944584548473358}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-04-12 02:55:52,031 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for c9ca1dc933715e878706f11781755586: 2023-04-12 02:55:52,031 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testSplitTableRegion,41,1681268150430.c9ca1dc933715e878706f11781755586., pid=149, masterSystemTime=1681268151966 2023-04-12 02:55:52,031 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: Opening Region; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 02:55:52,032 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: Opening Region; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 02:55:52,032 DEBUG [RS:0;jenkins-hbase4:34579-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-12 02:55:52,032 DEBUG [RS:0;jenkins-hbase4:34579-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-12 02:55:52,032 INFO [RS:0;jenkins-hbase4:34579-shortCompactions-0] regionserver.HStore(1898): Keeping/Overriding Compaction request priority to -2147482648 for CF cf1 since it belongs to recently split daughter region testSplitTableRegion,41,1681268150430.c9ca1dc933715e878706f11781755586. 2023-04-12 02:55:52,032 DEBUG [RS:0;jenkins-hbase4:34579-shortCompactions-0] regionserver.HStore(1912): c9ca1dc933715e878706f11781755586/cf1 is initiating minor compaction (all files) 2023-04-12 02:55:52,032 INFO [RS:0;jenkins-hbase4:34579-longCompactions-0] regionserver.HStore(1898): Keeping/Overriding Compaction request priority to -2147482648 for CF cf2 since it belongs to recently split daughter region testSplitTableRegion,41,1681268150430.c9ca1dc933715e878706f11781755586. 2023-04-12 02:55:52,033 INFO [RS:0;jenkins-hbase4:34579-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of c9ca1dc933715e878706f11781755586/cf1 in testSplitTableRegion,41,1681268150430.c9ca1dc933715e878706f11781755586. 2023-04-12 02:55:52,033 DEBUG [RS:0;jenkins-hbase4:34579-longCompactions-0] regionserver.HStore(1912): c9ca1dc933715e878706f11781755586/cf2 is initiating minor compaction (all files) 2023-04-12 02:55:52,033 INFO [RS:0;jenkins-hbase4:34579-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c9ca1dc933715e878706f11781755586/cf2 in testSplitTableRegion,41,1681268150430.c9ca1dc933715e878706f11781755586. 2023-04-12 02:55:52,033 INFO [RS:0;jenkins-hbase4:34579-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c9ca1dc933715e878706f11781755586/cf1/bd8f39350eae4e18bebd947afb5d9f5e.c426152f6c5cdf7d09ab747442f08ea4->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/cf1/bd8f39350eae4e18bebd947afb5d9f5e-top, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c9ca1dc933715e878706f11781755586/cf1/72636f8bfb49470784b49ea8615a5c6a.c426152f6c5cdf7d09ab747442f08ea4->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/cf1/72636f8bfb49470784b49ea8615a5c6a-top, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c9ca1dc933715e878706f11781755586/cf1/3e014fee55b841d9a93f29d49aa55156.c426152f6c5cdf7d09ab747442f08ea4->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/cf1/3e014fee55b841d9a93f29d49aa55156-top, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c9ca1dc933715e878706f11781755586/cf1/52c29c1aa57f4576b791c97686dd2e0d.c426152f6c5cdf7d09ab747442f08ea4->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/cf1/52c29c1aa57f4576b791c97686dd2e0d-top, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c9ca1dc933715e878706f11781755586/cf1/a54d3a0a458d49dbbc18bbd89c01e473.c426152f6c5cdf7d09ab747442f08ea4->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/cf1/a54d3a0a458d49dbbc18bbd89c01e473-top, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c9ca1dc933715e878706f11781755586/cf1/79de355eab9c46f98c089a49fcf5f846.c426152f6c5cdf7d09ab747442f08ea4->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/cf1/79de355eab9c46f98c089a49fcf5f846-top, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c9ca1dc933715e878706f11781755586/cf1/1f290c1ae6ff4ac9aa79371af93b04cf.c426152f6c5cdf7d09ab747442f08ea4->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/cf1/1f290c1ae6ff4ac9aa79371af93b04cf-top] into tmpdir=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c9ca1dc933715e878706f11781755586/.tmp, totalSize=34.9 K 2023-04-12 02:55:52,033 INFO [RS:0;jenkins-hbase4:34579-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c9ca1dc933715e878706f11781755586/cf2/ba426c6436064f9ca5ca16cecc0d8c98.c426152f6c5cdf7d09ab747442f08ea4->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/cf2/ba426c6436064f9ca5ca16cecc0d8c98-top, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c9ca1dc933715e878706f11781755586/cf2/f5ea9d0c0a4c491f91c40210e3a13892.c426152f6c5cdf7d09ab747442f08ea4->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/cf2/f5ea9d0c0a4c491f91c40210e3a13892-top, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c9ca1dc933715e878706f11781755586/cf2/c4a22b15c4714bc1a6f7b206d3315186.c426152f6c5cdf7d09ab747442f08ea4->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/cf2/c4a22b15c4714bc1a6f7b206d3315186-top, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c9ca1dc933715e878706f11781755586/cf2/e4c7205c2e024488b0db4154888f3f50.c426152f6c5cdf7d09ab747442f08ea4->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/cf2/e4c7205c2e024488b0db4154888f3f50-top, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c9ca1dc933715e878706f11781755586/cf2/e7cb094353e04c1c89a6b020574e3da1.c426152f6c5cdf7d09ab747442f08ea4->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/cf2/e7cb094353e04c1c89a6b020574e3da1-top, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c9ca1dc933715e878706f11781755586/cf2/506165cd013640888b0ebd392bd533ab.c426152f6c5cdf7d09ab747442f08ea4->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/cf2/506165cd013640888b0ebd392bd533ab-top, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c9ca1dc933715e878706f11781755586/cf2/5baff79a42ff479888e579a42af26748.c426152f6c5cdf7d09ab747442f08ea4->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/cf2/5baff79a42ff479888e579a42af26748-top] into tmpdir=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c9ca1dc933715e878706f11781755586/.tmp, totalSize=34.9 K 2023-04-12 02:55:52,033 DEBUG [RS:0;jenkins-hbase4:34579-shortCompactions-0] compactions.Compactor(207): Compacting bd8f39350eae4e18bebd947afb5d9f5e.c426152f6c5cdf7d09ab747442f08ea4, keycount=1, bloomtype=ROW, size=4.8 K, encoding=NONE, compression=NONE, seqNum=7, earliestPutTs=1681268148854 2023-04-12 02:55:52,034 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testSplitTableRegion,41,1681268150430.c9ca1dc933715e878706f11781755586. 2023-04-12 02:55:52,034 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testSplitTableRegion,41,1681268150430.c9ca1dc933715e878706f11781755586. 2023-04-12 02:55:52,034 DEBUG [RS:0;jenkins-hbase4:34579-longCompactions-0] compactions.Compactor(207): Compacting ba426c6436064f9ca5ca16cecc0d8c98.c426152f6c5cdf7d09ab747442f08ea4, keycount=1, bloomtype=ROW, size=4.8 K, encoding=NONE, compression=NONE, seqNum=7, earliestPutTs=1681268148854 2023-04-12 02:55:52,034 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testSplitTableRegion,,1681268150430.9a391ed3cdb851dca0e4ca65b3a980e8. 2023-04-12 02:55:52,034 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 9a391ed3cdb851dca0e4ca65b3a980e8, NAME => 'testSplitTableRegion,,1681268150430.9a391ed3cdb851dca0e4ca65b3a980e8.', STARTKEY => '', ENDKEY => '41'} 2023-04-12 02:55:52,034 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=148 updating hbase:meta row=c9ca1dc933715e878706f11781755586, regionState=OPEN, openSeqNum=86, regionLocation=jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:52,034 DEBUG [RS:0;jenkins-hbase4:34579-shortCompactions-0] compactions.Compactor(207): Compacting 72636f8bfb49470784b49ea8615a5c6a.c426152f6c5cdf7d09ab747442f08ea4, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=20, earliestPutTs=1681268149058 2023-04-12 02:55:52,034 DEBUG [RS:0;jenkins-hbase4:34579-longCompactions-0] compactions.Compactor(207): Compacting f5ea9d0c0a4c491f91c40210e3a13892.c426152f6c5cdf7d09ab747442f08ea4, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=20, earliestPutTs=1681268149058 2023-04-12 02:55:52,034 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(174): System coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro loaded, priority=536870911. 2023-04-12 02:55:52,034 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testSplitTableRegion,41,1681268150430.c9ca1dc933715e878706f11781755586.","families":{"info":[{"qualifier":"regioninfo","vlen":56,"tag":[],"timestamp":"1681268152034"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1681268152034"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681268152034"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681268152034"}]},"ts":"1681268152034"} 2023-04-12 02:55:52,034 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testSplitTableRegion 9a391ed3cdb851dca0e4ca65b3a980e8 2023-04-12 02:55:52,034 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated testSplitTableRegion,,1681268150430.9a391ed3cdb851dca0e4ca65b3a980e8.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-12 02:55:52,034 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 9a391ed3cdb851dca0e4ca65b3a980e8 2023-04-12 02:55:52,034 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 9a391ed3cdb851dca0e4ca65b3a980e8 2023-04-12 02:55:52,034 DEBUG [RS:0;jenkins-hbase4:34579-shortCompactions-0] compactions.Compactor(207): Compacting 3e014fee55b841d9a93f29d49aa55156.c426152f6c5cdf7d09ab747442f08ea4, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=33, earliestPutTs=1681268149674 2023-04-12 02:55:52,034 DEBUG [RS:0;jenkins-hbase4:34579-longCompactions-0] compactions.Compactor(207): Compacting c4a22b15c4714bc1a6f7b206d3315186.c426152f6c5cdf7d09ab747442f08ea4, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=33, earliestPutTs=1681268149674 2023-04-12 02:55:52,035 DEBUG [RS:0;jenkins-hbase4:34579-shortCompactions-0] compactions.Compactor(207): Compacting 52c29c1aa57f4576b791c97686dd2e0d.c426152f6c5cdf7d09ab747442f08ea4, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=46, earliestPutTs=1681268149865 2023-04-12 02:55:52,035 DEBUG [RS:0;jenkins-hbase4:34579-longCompactions-0] compactions.Compactor(207): Compacting e4c7205c2e024488b0db4154888f3f50.c426152f6c5cdf7d09ab747442f08ea4, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=46, earliestPutTs=1681268149865 2023-04-12 02:55:52,035 DEBUG [RS:0;jenkins-hbase4:34579-shortCompactions-0] compactions.Compactor(207): Compacting a54d3a0a458d49dbbc18bbd89c01e473.c426152f6c5cdf7d09ab747442f08ea4, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=59, earliestPutTs=1681268150056 2023-04-12 02:55:52,036 INFO [StoreOpener-9a391ed3cdb851dca0e4ca65b3a980e8-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf1 of region 9a391ed3cdb851dca0e4ca65b3a980e8 2023-04-12 02:55:52,036 DEBUG [RS:0;jenkins-hbase4:34579-longCompactions-0] compactions.Compactor(207): Compacting e7cb094353e04c1c89a6b020574e3da1.c426152f6c5cdf7d09ab747442f08ea4, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=59, earliestPutTs=1681268150056 2023-04-12 02:55:52,036 DEBUG [RS:0;jenkins-hbase4:34579-shortCompactions-0] compactions.Compactor(207): Compacting 79de355eab9c46f98c089a49fcf5f846.c426152f6c5cdf7d09ab747442f08ea4, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=72, earliestPutTs=1681268150256 2023-04-12 02:55:52,036 DEBUG [RS:0;jenkins-hbase4:34579-shortCompactions-0] compactions.Compactor(207): Compacting 1f290c1ae6ff4ac9aa79371af93b04cf.c426152f6c5cdf7d09ab747442f08ea4, keycount=4, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=83, earliestPutTs=1681268150424 2023-04-12 02:55:52,036 DEBUG [RS:0;jenkins-hbase4:34579-longCompactions-0] compactions.Compactor(207): Compacting 506165cd013640888b0ebd392bd533ab.c426152f6c5cdf7d09ab747442f08ea4, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=72, earliestPutTs=1681268150256 2023-04-12 02:55:52,037 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=149, resume processing ppid=148 2023-04-12 02:55:52,037 DEBUG [StoreOpener-9a391ed3cdb851dca0e4ca65b3a980e8-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/9a391ed3cdb851dca0e4ca65b3a980e8/cf1 2023-04-12 02:55:52,037 DEBUG [StoreOpener-9a391ed3cdb851dca0e4ca65b3a980e8-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/9a391ed3cdb851dca0e4ca65b3a980e8/cf1 2023-04-12 02:55:52,037 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=149, ppid=148, state=SUCCESS; OpenRegionProcedure c9ca1dc933715e878706f11781755586, server=jenkins-hbase4.apache.org,34579,1681268031719 in 220 msec 2023-04-12 02:55:52,037 DEBUG [RS:0;jenkins-hbase4:34579-longCompactions-0] compactions.Compactor(207): Compacting 5baff79a42ff479888e579a42af26748.c426152f6c5cdf7d09ab747442f08ea4, keycount=4, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=83, earliestPutTs=1681268150424 2023-04-12 02:55:52,037 INFO [StoreOpener-9a391ed3cdb851dca0e4ca65b3a980e8-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 9a391ed3cdb851dca0e4ca65b3a980e8 columnFamilyName cf1 2023-04-12 02:55:52,038 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=148, ppid=144, state=SUCCESS; TransitRegionStateProcedure table=testSplitTableRegion, region=c9ca1dc933715e878706f11781755586, ASSIGN in 378 msec 2023-04-12 02:55:52,048 INFO [RS:0;jenkins-hbase4:34579-shortCompactions-0] throttle.PressureAwareThroughputController(145): c9ca1dc933715e878706f11781755586#cf1#compaction#142 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 02:55:52,048 INFO [RS:0;jenkins-hbase4:34579-longCompactions-0] throttle.PressureAwareThroughputController(145): c9ca1dc933715e878706f11781755586#cf2#compaction#143 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 02:55:52,050 DEBUG [StoreOpener-9a391ed3cdb851dca0e4ca65b3a980e8-1] regionserver.HStore(539): loaded hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/9a391ed3cdb851dca0e4ca65b3a980e8/cf1/1f290c1ae6ff4ac9aa79371af93b04cf.c426152f6c5cdf7d09ab747442f08ea4->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/cf1/1f290c1ae6ff4ac9aa79371af93b04cf-bottom 2023-04-12 02:55:52,053 DEBUG [StoreOpener-9a391ed3cdb851dca0e4ca65b3a980e8-1] regionserver.HStore(539): loaded hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/9a391ed3cdb851dca0e4ca65b3a980e8/cf1/3e014fee55b841d9a93f29d49aa55156.c426152f6c5cdf7d09ab747442f08ea4->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/cf1/3e014fee55b841d9a93f29d49aa55156-bottom 2023-04-12 02:55:52,055 DEBUG [StoreOpener-9a391ed3cdb851dca0e4ca65b3a980e8-1] regionserver.HStore(539): loaded hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/9a391ed3cdb851dca0e4ca65b3a980e8/cf1/52c29c1aa57f4576b791c97686dd2e0d.c426152f6c5cdf7d09ab747442f08ea4->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/cf1/52c29c1aa57f4576b791c97686dd2e0d-bottom 2023-04-12 02:55:52,064 DEBUG [StoreOpener-9a391ed3cdb851dca0e4ca65b3a980e8-1] regionserver.HStore(539): loaded hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/9a391ed3cdb851dca0e4ca65b3a980e8/cf1/72636f8bfb49470784b49ea8615a5c6a.c426152f6c5cdf7d09ab747442f08ea4->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/cf1/72636f8bfb49470784b49ea8615a5c6a-bottom 2023-04-12 02:55:52,068 DEBUG [RS:0;jenkins-hbase4:34579-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c9ca1dc933715e878706f11781755586/.tmp/cf1/3199801c87964457a97c9936a9f2221d as hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c9ca1dc933715e878706f11781755586/cf1/3199801c87964457a97c9936a9f2221d 2023-04-12 02:55:52,068 DEBUG [StoreOpener-9a391ed3cdb851dca0e4ca65b3a980e8-1] regionserver.HStore(539): loaded hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/9a391ed3cdb851dca0e4ca65b3a980e8/cf1/79de355eab9c46f98c089a49fcf5f846.c426152f6c5cdf7d09ab747442f08ea4->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/cf1/79de355eab9c46f98c089a49fcf5f846-bottom 2023-04-12 02:55:52,069 DEBUG [RS:0;jenkins-hbase4:34579-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c9ca1dc933715e878706f11781755586/.tmp/cf2/7745c9273e724bbb91b3a05cbeb0bf13 as hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c9ca1dc933715e878706f11781755586/cf2/7745c9273e724bbb91b3a05cbeb0bf13 2023-04-12 02:55:52,072 DEBUG [StoreOpener-9a391ed3cdb851dca0e4ca65b3a980e8-1] regionserver.HStore(539): loaded hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/9a391ed3cdb851dca0e4ca65b3a980e8/cf1/a54d3a0a458d49dbbc18bbd89c01e473.c426152f6c5cdf7d09ab747442f08ea4->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/cf1/a54d3a0a458d49dbbc18bbd89c01e473-bottom 2023-04-12 02:55:52,072 INFO [RS:0;jenkins-hbase4:34579-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in c9ca1dc933715e878706f11781755586/cf1 of c9ca1dc933715e878706f11781755586 into 3199801c87964457a97c9936a9f2221d(size=6.1 K), total size for store is 6.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 02:55:52,072 DEBUG [RS:0;jenkins-hbase4:34579-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for c9ca1dc933715e878706f11781755586: 2023-04-12 02:55:52,072 INFO [RS:0;jenkins-hbase4:34579-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=testSplitTableRegion,41,1681268150430.c9ca1dc933715e878706f11781755586., storeName=c9ca1dc933715e878706f11781755586/cf1, priority=9, startTime=1681268152031; duration=0sec 2023-04-12 02:55:52,072 DEBUG [RS:0;jenkins-hbase4:34579-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 02:55:52,073 INFO [RS:0;jenkins-hbase4:34579-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in c9ca1dc933715e878706f11781755586/cf2 of c9ca1dc933715e878706f11781755586 into 7745c9273e724bbb91b3a05cbeb0bf13(size=6.1 K), total size for store is 6.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 02:55:52,074 DEBUG [RS:0;jenkins-hbase4:34579-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c9ca1dc933715e878706f11781755586: 2023-04-12 02:55:52,074 INFO [RS:0;jenkins-hbase4:34579-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=testSplitTableRegion,41,1681268150430.c9ca1dc933715e878706f11781755586., storeName=c9ca1dc933715e878706f11781755586/cf2, priority=9, startTime=1681268152032; duration=0sec 2023-04-12 02:55:52,074 DEBUG [RS:0;jenkins-hbase4:34579-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 02:55:52,075 DEBUG [StoreOpener-9a391ed3cdb851dca0e4ca65b3a980e8-1] regionserver.HStore(539): loaded hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/9a391ed3cdb851dca0e4ca65b3a980e8/cf1/bd8f39350eae4e18bebd947afb5d9f5e.c426152f6c5cdf7d09ab747442f08ea4->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/cf1/bd8f39350eae4e18bebd947afb5d9f5e-bottom 2023-04-12 02:55:52,075 INFO [StoreOpener-9a391ed3cdb851dca0e4ca65b3a980e8-1] regionserver.HStore(310): Store=9a391ed3cdb851dca0e4ca65b3a980e8/cf1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-12 02:55:52,075 INFO [StoreOpener-9a391ed3cdb851dca0e4ca65b3a980e8-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf2 of region 9a391ed3cdb851dca0e4ca65b3a980e8 2023-04-12 02:55:52,076 DEBUG [StoreOpener-9a391ed3cdb851dca0e4ca65b3a980e8-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/9a391ed3cdb851dca0e4ca65b3a980e8/cf2 2023-04-12 02:55:52,076 DEBUG [StoreOpener-9a391ed3cdb851dca0e4ca65b3a980e8-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/9a391ed3cdb851dca0e4ca65b3a980e8/cf2 2023-04-12 02:55:52,076 INFO [StoreOpener-9a391ed3cdb851dca0e4ca65b3a980e8-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 9a391ed3cdb851dca0e4ca65b3a980e8 columnFamilyName cf2 2023-04-12 02:55:52,085 DEBUG [StoreOpener-9a391ed3cdb851dca0e4ca65b3a980e8-1] regionserver.HStore(539): loaded hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/9a391ed3cdb851dca0e4ca65b3a980e8/cf2/506165cd013640888b0ebd392bd533ab.c426152f6c5cdf7d09ab747442f08ea4->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/cf2/506165cd013640888b0ebd392bd533ab-bottom 2023-04-12 02:55:52,087 DEBUG [StoreOpener-9a391ed3cdb851dca0e4ca65b3a980e8-1] regionserver.HStore(539): loaded hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/9a391ed3cdb851dca0e4ca65b3a980e8/cf2/5baff79a42ff479888e579a42af26748.c426152f6c5cdf7d09ab747442f08ea4->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/cf2/5baff79a42ff479888e579a42af26748-bottom 2023-04-12 02:55:52,090 DEBUG [StoreOpener-9a391ed3cdb851dca0e4ca65b3a980e8-1] regionserver.HStore(539): loaded hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/9a391ed3cdb851dca0e4ca65b3a980e8/cf2/ba426c6436064f9ca5ca16cecc0d8c98.c426152f6c5cdf7d09ab747442f08ea4->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/cf2/ba426c6436064f9ca5ca16cecc0d8c98-bottom 2023-04-12 02:55:52,092 DEBUG [StoreOpener-9a391ed3cdb851dca0e4ca65b3a980e8-1] regionserver.HStore(539): loaded hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/9a391ed3cdb851dca0e4ca65b3a980e8/cf2/c4a22b15c4714bc1a6f7b206d3315186.c426152f6c5cdf7d09ab747442f08ea4->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/cf2/c4a22b15c4714bc1a6f7b206d3315186-bottom 2023-04-12 02:55:52,095 DEBUG [StoreOpener-9a391ed3cdb851dca0e4ca65b3a980e8-1] regionserver.HStore(539): loaded hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/9a391ed3cdb851dca0e4ca65b3a980e8/cf2/e4c7205c2e024488b0db4154888f3f50.c426152f6c5cdf7d09ab747442f08ea4->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/cf2/e4c7205c2e024488b0db4154888f3f50-bottom 2023-04-12 02:55:52,097 DEBUG [StoreOpener-9a391ed3cdb851dca0e4ca65b3a980e8-1] regionserver.HStore(539): loaded hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/9a391ed3cdb851dca0e4ca65b3a980e8/cf2/e7cb094353e04c1c89a6b020574e3da1.c426152f6c5cdf7d09ab747442f08ea4->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/cf2/e7cb094353e04c1c89a6b020574e3da1-bottom 2023-04-12 02:55:52,100 DEBUG [StoreOpener-9a391ed3cdb851dca0e4ca65b3a980e8-1] regionserver.HStore(539): loaded hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/9a391ed3cdb851dca0e4ca65b3a980e8/cf2/f5ea9d0c0a4c491f91c40210e3a13892.c426152f6c5cdf7d09ab747442f08ea4->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/cf2/f5ea9d0c0a4c491f91c40210e3a13892-bottom 2023-04-12 02:55:52,100 INFO [StoreOpener-9a391ed3cdb851dca0e4ca65b3a980e8-1] regionserver.HStore(310): Store=9a391ed3cdb851dca0e4ca65b3a980e8/cf2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-12 02:55:52,100 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/9a391ed3cdb851dca0e4ca65b3a980e8 2023-04-12 02:55:52,101 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/9a391ed3cdb851dca0e4ca65b3a980e8 2023-04-12 02:55:52,102 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testSplitTableRegion descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-04-12 02:55:52,103 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 9a391ed3cdb851dca0e4ca65b3a980e8 2023-04-12 02:55:52,103 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 9a391ed3cdb851dca0e4ca65b3a980e8; next sequenceid=86; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=59573942, jitterRate=-0.11227908730506897}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-04-12 02:55:52,103 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 9a391ed3cdb851dca0e4ca65b3a980e8: 2023-04-12 02:55:52,104 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testSplitTableRegion,,1681268150430.9a391ed3cdb851dca0e4ca65b3a980e8., pid=150, masterSystemTime=1681268151966 2023-04-12 02:55:52,104 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: Opening Region; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 02:55:52,105 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: Opening Region; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-12 02:55:52,105 DEBUG [RS:0;jenkins-hbase4:34579-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-12 02:55:52,105 DEBUG [RS:0;jenkins-hbase4:34579-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-12 02:55:52,105 INFO [RS:0;jenkins-hbase4:34579-shortCompactions-0] regionserver.HStore(1898): Keeping/Overriding Compaction request priority to -2147482648 for CF cf1 since it belongs to recently split daughter region testSplitTableRegion,,1681268150430.9a391ed3cdb851dca0e4ca65b3a980e8. 2023-04-12 02:55:52,105 INFO [RS:0;jenkins-hbase4:34579-longCompactions-0] regionserver.HStore(1898): Keeping/Overriding Compaction request priority to -2147482648 for CF cf2 since it belongs to recently split daughter region testSplitTableRegion,,1681268150430.9a391ed3cdb851dca0e4ca65b3a980e8. 2023-04-12 02:55:52,105 DEBUG [RS:0;jenkins-hbase4:34579-shortCompactions-0] regionserver.HStore(1912): 9a391ed3cdb851dca0e4ca65b3a980e8/cf1 is initiating minor compaction (all files) 2023-04-12 02:55:52,105 DEBUG [RS:0;jenkins-hbase4:34579-longCompactions-0] regionserver.HStore(1912): 9a391ed3cdb851dca0e4ca65b3a980e8/cf2 is initiating minor compaction (all files) 2023-04-12 02:55:52,105 INFO [RS:0;jenkins-hbase4:34579-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 9a391ed3cdb851dca0e4ca65b3a980e8/cf1 in testSplitTableRegion,,1681268150430.9a391ed3cdb851dca0e4ca65b3a980e8. 2023-04-12 02:55:52,105 INFO [RS:0;jenkins-hbase4:34579-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 9a391ed3cdb851dca0e4ca65b3a980e8/cf2 in testSplitTableRegion,,1681268150430.9a391ed3cdb851dca0e4ca65b3a980e8. 2023-04-12 02:55:52,105 INFO [RS:0;jenkins-hbase4:34579-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/9a391ed3cdb851dca0e4ca65b3a980e8/cf1/bd8f39350eae4e18bebd947afb5d9f5e.c426152f6c5cdf7d09ab747442f08ea4->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/cf1/bd8f39350eae4e18bebd947afb5d9f5e-bottom, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/9a391ed3cdb851dca0e4ca65b3a980e8/cf1/72636f8bfb49470784b49ea8615a5c6a.c426152f6c5cdf7d09ab747442f08ea4->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/cf1/72636f8bfb49470784b49ea8615a5c6a-bottom, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/9a391ed3cdb851dca0e4ca65b3a980e8/cf1/3e014fee55b841d9a93f29d49aa55156.c426152f6c5cdf7d09ab747442f08ea4->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/cf1/3e014fee55b841d9a93f29d49aa55156-bottom, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/9a391ed3cdb851dca0e4ca65b3a980e8/cf1/52c29c1aa57f4576b791c97686dd2e0d.c426152f6c5cdf7d09ab747442f08ea4->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/cf1/52c29c1aa57f4576b791c97686dd2e0d-bottom, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/9a391ed3cdb851dca0e4ca65b3a980e8/cf1/a54d3a0a458d49dbbc18bbd89c01e473.c426152f6c5cdf7d09ab747442f08ea4->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/cf1/a54d3a0a458d49dbbc18bbd89c01e473-bottom, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/9a391ed3cdb851dca0e4ca65b3a980e8/cf1/79de355eab9c46f98c089a49fcf5f846.c426152f6c5cdf7d09ab747442f08ea4->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/cf1/79de355eab9c46f98c089a49fcf5f846-bottom, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/9a391ed3cdb851dca0e4ca65b3a980e8/cf1/1f290c1ae6ff4ac9aa79371af93b04cf.c426152f6c5cdf7d09ab747442f08ea4->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/cf1/1f290c1ae6ff4ac9aa79371af93b04cf-bottom] into tmpdir=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/9a391ed3cdb851dca0e4ca65b3a980e8/.tmp, totalSize=34.9 K 2023-04-12 02:55:52,106 INFO [RS:0;jenkins-hbase4:34579-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/9a391ed3cdb851dca0e4ca65b3a980e8/cf2/ba426c6436064f9ca5ca16cecc0d8c98.c426152f6c5cdf7d09ab747442f08ea4->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/cf2/ba426c6436064f9ca5ca16cecc0d8c98-bottom, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/9a391ed3cdb851dca0e4ca65b3a980e8/cf2/f5ea9d0c0a4c491f91c40210e3a13892.c426152f6c5cdf7d09ab747442f08ea4->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/cf2/f5ea9d0c0a4c491f91c40210e3a13892-bottom, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/9a391ed3cdb851dca0e4ca65b3a980e8/cf2/c4a22b15c4714bc1a6f7b206d3315186.c426152f6c5cdf7d09ab747442f08ea4->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/cf2/c4a22b15c4714bc1a6f7b206d3315186-bottom, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/9a391ed3cdb851dca0e4ca65b3a980e8/cf2/e4c7205c2e024488b0db4154888f3f50.c426152f6c5cdf7d09ab747442f08ea4->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/cf2/e4c7205c2e024488b0db4154888f3f50-bottom, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/9a391ed3cdb851dca0e4ca65b3a980e8/cf2/e7cb094353e04c1c89a6b020574e3da1.c426152f6c5cdf7d09ab747442f08ea4->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/cf2/e7cb094353e04c1c89a6b020574e3da1-bottom, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/9a391ed3cdb851dca0e4ca65b3a980e8/cf2/506165cd013640888b0ebd392bd533ab.c426152f6c5cdf7d09ab747442f08ea4->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/cf2/506165cd013640888b0ebd392bd533ab-bottom, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/9a391ed3cdb851dca0e4ca65b3a980e8/cf2/5baff79a42ff479888e579a42af26748.c426152f6c5cdf7d09ab747442f08ea4->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/cf2/5baff79a42ff479888e579a42af26748-bottom] into tmpdir=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/9a391ed3cdb851dca0e4ca65b3a980e8/.tmp, totalSize=34.9 K 2023-04-12 02:55:52,106 DEBUG [RS:0;jenkins-hbase4:34579-shortCompactions-0] compactions.Compactor(207): Compacting bd8f39350eae4e18bebd947afb5d9f5e.c426152f6c5cdf7d09ab747442f08ea4, keycount=1, bloomtype=ROW, size=4.8 K, encoding=NONE, compression=NONE, seqNum=6, earliestPutTs=1681268148854 2023-04-12 02:55:52,106 DEBUG [RS:0;jenkins-hbase4:34579-longCompactions-0] compactions.Compactor(207): Compacting ba426c6436064f9ca5ca16cecc0d8c98.c426152f6c5cdf7d09ab747442f08ea4, keycount=1, bloomtype=ROW, size=4.8 K, encoding=NONE, compression=NONE, seqNum=6, earliestPutTs=1681268148854 2023-04-12 02:55:52,106 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testSplitTableRegion,,1681268150430.9a391ed3cdb851dca0e4ca65b3a980e8. 2023-04-12 02:55:52,106 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testSplitTableRegion,,1681268150430.9a391ed3cdb851dca0e4ca65b3a980e8. 2023-04-12 02:55:52,106 DEBUG [RS:0;jenkins-hbase4:34579-shortCompactions-0] compactions.Compactor(207): Compacting 72636f8bfb49470784b49ea8615a5c6a.c426152f6c5cdf7d09ab747442f08ea4, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=19, earliestPutTs=1681268149058 2023-04-12 02:55:52,107 DEBUG [RS:0;jenkins-hbase4:34579-longCompactions-0] compactions.Compactor(207): Compacting f5ea9d0c0a4c491f91c40210e3a13892.c426152f6c5cdf7d09ab747442f08ea4, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=19, earliestPutTs=1681268149058 2023-04-12 02:55:52,107 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=147 updating hbase:meta row=9a391ed3cdb851dca0e4ca65b3a980e8, regionState=OPEN, openSeqNum=86, regionLocation=jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:52,107 DEBUG [RS:0;jenkins-hbase4:34579-shortCompactions-0] compactions.Compactor(207): Compacting 3e014fee55b841d9a93f29d49aa55156.c426152f6c5cdf7d09ab747442f08ea4, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=32, earliestPutTs=1681268149674 2023-04-12 02:55:52,107 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testSplitTableRegion,,1681268150430.9a391ed3cdb851dca0e4ca65b3a980e8.","families":{"info":[{"qualifier":"regioninfo","vlen":56,"tag":[],"timestamp":"1681268152107"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1681268152107"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1681268152107"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1681268152107"}]},"ts":"1681268152107"} 2023-04-12 02:55:52,107 DEBUG [RS:0;jenkins-hbase4:34579-longCompactions-0] compactions.Compactor(207): Compacting c4a22b15c4714bc1a6f7b206d3315186.c426152f6c5cdf7d09ab747442f08ea4, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=32, earliestPutTs=1681268149674 2023-04-12 02:55:52,107 DEBUG [RS:0;jenkins-hbase4:34579-shortCompactions-0] compactions.Compactor(207): Compacting 52c29c1aa57f4576b791c97686dd2e0d.c426152f6c5cdf7d09ab747442f08ea4, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=45, earliestPutTs=1681268149865 2023-04-12 02:55:52,107 DEBUG [RS:0;jenkins-hbase4:34579-longCompactions-0] compactions.Compactor(207): Compacting e4c7205c2e024488b0db4154888f3f50.c426152f6c5cdf7d09ab747442f08ea4, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=45, earliestPutTs=1681268149865 2023-04-12 02:55:52,107 DEBUG [RS:0;jenkins-hbase4:34579-shortCompactions-0] compactions.Compactor(207): Compacting a54d3a0a458d49dbbc18bbd89c01e473.c426152f6c5cdf7d09ab747442f08ea4, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=58, earliestPutTs=1681268150056 2023-04-12 02:55:52,108 DEBUG [RS:0;jenkins-hbase4:34579-longCompactions-0] compactions.Compactor(207): Compacting e7cb094353e04c1c89a6b020574e3da1.c426152f6c5cdf7d09ab747442f08ea4, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=58, earliestPutTs=1681268150056 2023-04-12 02:55:52,108 DEBUG [RS:0;jenkins-hbase4:34579-shortCompactions-0] compactions.Compactor(207): Compacting 79de355eab9c46f98c089a49fcf5f846.c426152f6c5cdf7d09ab747442f08ea4, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=71, earliestPutTs=1681268150256 2023-04-12 02:55:52,108 DEBUG [RS:0;jenkins-hbase4:34579-longCompactions-0] compactions.Compactor(207): Compacting 506165cd013640888b0ebd392bd533ab.c426152f6c5cdf7d09ab747442f08ea4, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=71, earliestPutTs=1681268150256 2023-04-12 02:55:52,108 DEBUG [RS:0;jenkins-hbase4:34579-shortCompactions-0] compactions.Compactor(207): Compacting 1f290c1ae6ff4ac9aa79371af93b04cf.c426152f6c5cdf7d09ab747442f08ea4, keycount=4, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=82, earliestPutTs=1681268150424 2023-04-12 02:55:52,108 DEBUG [RS:0;jenkins-hbase4:34579-longCompactions-0] compactions.Compactor(207): Compacting 5baff79a42ff479888e579a42af26748.c426152f6c5cdf7d09ab747442f08ea4, keycount=4, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=82, earliestPutTs=1681268150424 2023-04-12 02:55:52,109 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=150, resume processing ppid=147 2023-04-12 02:55:52,109 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=150, ppid=147, state=SUCCESS; OpenRegionProcedure 9a391ed3cdb851dca0e4ca65b3a980e8, server=jenkins-hbase4.apache.org,34579,1681268031719 in 290 msec 2023-04-12 02:55:52,111 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=147, resume processing ppid=144 2023-04-12 02:55:52,111 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=147, ppid=144, state=SUCCESS; TransitRegionStateProcedure table=testSplitTableRegion, region=9a391ed3cdb851dca0e4ca65b3a980e8, ASSIGN in 450 msec 2023-04-12 02:55:52,112 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=144, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegion, parent=c426152f6c5cdf7d09ab747442f08ea4, daughterA=9a391ed3cdb851dca0e4ca65b3a980e8, daughterB=c9ca1dc933715e878706f11781755586 in 1.6810 sec 2023-04-12 02:55:52,117 INFO [RS:0;jenkins-hbase4:34579-shortCompactions-0] throttle.PressureAwareThroughputController(145): 9a391ed3cdb851dca0e4ca65b3a980e8#cf1#compaction#144 average throughput is 0.86 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 02:55:52,117 INFO [RS:0;jenkins-hbase4:34579-longCompactions-0] throttle.PressureAwareThroughputController(145): 9a391ed3cdb851dca0e4ca65b3a980e8#cf2#compaction#145 average throughput is 0.86 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-12 02:55:52,129 DEBUG [RS:0;jenkins-hbase4:34579-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/9a391ed3cdb851dca0e4ca65b3a980e8/.tmp/cf2/f78639c04e9f489c9d0735c8ce312492 as hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/9a391ed3cdb851dca0e4ca65b3a980e8/cf2/f78639c04e9f489c9d0735c8ce312492 2023-04-12 02:55:52,129 DEBUG [RS:0;jenkins-hbase4:34579-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/9a391ed3cdb851dca0e4ca65b3a980e8/.tmp/cf1/815b43de660541e097ab58c3b0dcf022 as hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/9a391ed3cdb851dca0e4ca65b3a980e8/cf1/815b43de660541e097ab58c3b0dcf022 2023-04-12 02:55:52,134 INFO [RS:0;jenkins-hbase4:34579-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 9a391ed3cdb851dca0e4ca65b3a980e8/cf2 of 9a391ed3cdb851dca0e4ca65b3a980e8 into f78639c04e9f489c9d0735c8ce312492(size=6.1 K), total size for store is 6.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 02:55:52,134 INFO [RS:0;jenkins-hbase4:34579-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 9a391ed3cdb851dca0e4ca65b3a980e8/cf1 of 9a391ed3cdb851dca0e4ca65b3a980e8 into 815b43de660541e097ab58c3b0dcf022(size=6.1 K), total size for store is 6.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-12 02:55:52,134 DEBUG [RS:0;jenkins-hbase4:34579-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9a391ed3cdb851dca0e4ca65b3a980e8: 2023-04-12 02:55:52,134 DEBUG [RS:0;jenkins-hbase4:34579-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9a391ed3cdb851dca0e4ca65b3a980e8: 2023-04-12 02:55:52,134 INFO [RS:0;jenkins-hbase4:34579-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=testSplitTableRegion,,1681268150430.9a391ed3cdb851dca0e4ca65b3a980e8., storeName=9a391ed3cdb851dca0e4ca65b3a980e8/cf2, priority=9, startTime=1681268152105; duration=0sec 2023-04-12 02:55:52,134 INFO [RS:0;jenkins-hbase4:34579-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=testSplitTableRegion,,1681268150430.9a391ed3cdb851dca0e4ca65b3a980e8., storeName=9a391ed3cdb851dca0e4ca65b3a980e8/cf1, priority=9, startTime=1681268152104; duration=0sec 2023-04-12 02:55:52,134 DEBUG [RS:0;jenkins-hbase4:34579-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 02:55:52,134 DEBUG [RS:0;jenkins-hbase4:34579-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-12 02:55:52,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34579] ipc.CallRunner(144): callId: 1132 service: ClientService methodName: Scan size: 143 connection: 172.31.14.131:49302 deadline: 1681268212182, exception=org.apache.hadoop.hbase.NotServingRegionException: testSplitTableRegion,,1681268148351.c426152f6c5cdf7d09ab747442f08ea4. is not online on jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:52,248 ERROR [BP-1247239137-172.31.14.131-1681268026808 heartbeating to localhost/127.0.0.1:44691] datanode.BPServiceActor(848): Exception in BPOfferService for Block pool BP-1247239137-172.31.14.131-1681268026808 (Datanode Uuid ad316322-0da1-41fd-a03f-e3dd2b710c84) service to localhost/127.0.0.1:44691 java.lang.IllegalThreadStateException at java.lang.ThreadGroup.addUnstarted(ThreadGroup.java:867) at java.lang.Thread.init(Thread.java:407) at java.lang.Thread.init(Thread.java:351) at java.lang.Thread.(Thread.java:501) at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService$1.newThread(FsDatasetAsyncDiskService.java:102) at java.util.concurrent.ThreadPoolExecutor$Worker.(ThreadPoolExecutor.java:619) at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:932) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1367) at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService.execute(FsDatasetAsyncDiskService.java:172) at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService.deleteAsync(FsDatasetAsyncDiskService.java:221) at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.invalidate(FsDatasetImpl.java:2216) at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.invalidate(FsDatasetImpl.java:2140) at org.apache.hadoop.hdfs.server.datanode.BPOfferService.processCommandFromActive(BPOfferService.java:732) at org.apache.hadoop.hdfs.server.datanode.BPOfferService.processCommandFromActor(BPOfferService.java:679) at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.processCommand(BPServiceActor.java:880) at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:676) at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:846) at java.lang.Thread.run(Thread.java:750) 2023-04-12 02:55:52,291 DEBUG [Listener at localhost/46667] client.ClientScanner(240): Advancing internal scanner to startKey at '41', inclusive 2023-04-12 02:55:52,297 INFO [Listener at localhost/46667] assignment.TestSplitTableRegionProcedure(570): Row Count = 60 2023-04-12 02:55:52,298 DEBUG [Listener at localhost/46667] client.ClientScanner(240): Advancing internal scanner to startKey at '41', inclusive 2023-04-12 02:55:52,303 WARN [Listener at localhost/46667] procedure2.ProcedureTestingUtility(193): Set Kill before store update to: false 2023-04-12 02:55:52,304 INFO [Listener at localhost/46667] client.HBaseAdmin$15(890): Started disable of testSplitTableRegion 2023-04-12 02:55:52,305 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable testSplitTableRegion 2023-04-12 02:55:52,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] procedure2.ProcedureExecutor(1029): Stored pid=151, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testSplitTableRegion 2023-04-12 02:55:52,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] master.MasterRpcServices(1227): Checking to see if procedure is done pid=151 2023-04-12 02:55:52,308 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testSplitTableRegion","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681268152308"}]},"ts":"1681268152308"} 2023-04-12 02:55:52,308 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testSplitTableRegion, state=DISABLING in hbase:meta 2023-04-12 02:55:52,310 INFO [PEWorker-1] procedure.DisableTableProcedure(293): Set testSplitTableRegion to state=DISABLING 2023-04-12 02:55:52,311 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=152, ppid=151, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitTableRegion, region=9a391ed3cdb851dca0e4ca65b3a980e8, UNASSIGN}, {pid=153, ppid=151, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitTableRegion, region=c9ca1dc933715e878706f11781755586, UNASSIGN}] 2023-04-12 02:55:52,311 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=153, ppid=151, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitTableRegion, region=c9ca1dc933715e878706f11781755586, UNASSIGN 2023-04-12 02:55:52,312 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=153 updating hbase:meta row=c9ca1dc933715e878706f11781755586, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:52,312 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitTableRegion,41,1681268150430.c9ca1dc933715e878706f11781755586.","families":{"info":[{"qualifier":"regioninfo","vlen":56,"tag":[],"timestamp":"1681268152312"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681268152312"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681268152312"}]},"ts":"1681268152312"} 2023-04-12 02:55:52,313 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=154, ppid=153, state=RUNNABLE; CloseRegionProcedure c9ca1dc933715e878706f11781755586, server=jenkins-hbase4.apache.org,34579,1681268031719}] 2023-04-12 02:55:52,314 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=152, ppid=151, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitTableRegion, region=9a391ed3cdb851dca0e4ca65b3a980e8, UNASSIGN 2023-04-12 02:55:52,315 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=152 updating hbase:meta row=9a391ed3cdb851dca0e4ca65b3a980e8, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:52,315 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitTableRegion,,1681268150430.9a391ed3cdb851dca0e4ca65b3a980e8.","families":{"info":[{"qualifier":"regioninfo","vlen":56,"tag":[],"timestamp":"1681268152314"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1681268152314"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1681268152314"}]},"ts":"1681268152314"} 2023-04-12 02:55:52,317 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=155, ppid=152, state=RUNNABLE; CloseRegionProcedure 9a391ed3cdb851dca0e4ca65b3a980e8, server=jenkins-hbase4.apache.org,34579,1681268031719}] 2023-04-12 02:55:52,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] master.MasterRpcServices(1227): Checking to see if procedure is done pid=151 2023-04-12 02:55:52,464 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 9a391ed3cdb851dca0e4ca65b3a980e8 2023-04-12 02:55:52,464 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 9a391ed3cdb851dca0e4ca65b3a980e8, disabling compactions & flushes 2023-04-12 02:55:52,464 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region testSplitTableRegion,,1681268150430.9a391ed3cdb851dca0e4ca65b3a980e8. 2023-04-12 02:55:52,464 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testSplitTableRegion,,1681268150430.9a391ed3cdb851dca0e4ca65b3a980e8. 2023-04-12 02:55:52,464 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on testSplitTableRegion,,1681268150430.9a391ed3cdb851dca0e4ca65b3a980e8. after waiting 0 ms 2023-04-12 02:55:52,464 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region testSplitTableRegion,,1681268150430.9a391ed3cdb851dca0e4ca65b3a980e8. 2023-04-12 02:55:52,465 DEBUG [StoreCloser-testSplitTableRegion,,1681268150430.9a391ed3cdb851dca0e4ca65b3a980e8.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/9a391ed3cdb851dca0e4ca65b3a980e8/cf1/bd8f39350eae4e18bebd947afb5d9f5e.c426152f6c5cdf7d09ab747442f08ea4->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/cf1/bd8f39350eae4e18bebd947afb5d9f5e-bottom, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/9a391ed3cdb851dca0e4ca65b3a980e8/cf1/72636f8bfb49470784b49ea8615a5c6a.c426152f6c5cdf7d09ab747442f08ea4->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/cf1/72636f8bfb49470784b49ea8615a5c6a-bottom, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/9a391ed3cdb851dca0e4ca65b3a980e8/cf1/3e014fee55b841d9a93f29d49aa55156.c426152f6c5cdf7d09ab747442f08ea4->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/cf1/3e014fee55b841d9a93f29d49aa55156-bottom, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/9a391ed3cdb851dca0e4ca65b3a980e8/cf1/52c29c1aa57f4576b791c97686dd2e0d.c426152f6c5cdf7d09ab747442f08ea4->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/cf1/52c29c1aa57f4576b791c97686dd2e0d-bottom, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/9a391ed3cdb851dca0e4ca65b3a980e8/cf1/a54d3a0a458d49dbbc18bbd89c01e473.c426152f6c5cdf7d09ab747442f08ea4->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/cf1/a54d3a0a458d49dbbc18bbd89c01e473-bottom, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/9a391ed3cdb851dca0e4ca65b3a980e8/cf1/79de355eab9c46f98c089a49fcf5f846.c426152f6c5cdf7d09ab747442f08ea4->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/cf1/79de355eab9c46f98c089a49fcf5f846-bottom, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/9a391ed3cdb851dca0e4ca65b3a980e8/cf1/1f290c1ae6ff4ac9aa79371af93b04cf.c426152f6c5cdf7d09ab747442f08ea4->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/cf1/1f290c1ae6ff4ac9aa79371af93b04cf-bottom] to archive 2023-04-12 02:55:52,466 DEBUG [StoreCloser-testSplitTableRegion,,1681268150430.9a391ed3cdb851dca0e4ca65b3a980e8.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-12 02:55:52,467 DEBUG [StoreCloser-testSplitTableRegion,,1681268150430.9a391ed3cdb851dca0e4ca65b3a980e8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/9a391ed3cdb851dca0e4ca65b3a980e8/cf1/bd8f39350eae4e18bebd947afb5d9f5e.c426152f6c5cdf7d09ab747442f08ea4 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegion/9a391ed3cdb851dca0e4ca65b3a980e8/cf1/bd8f39350eae4e18bebd947afb5d9f5e.c426152f6c5cdf7d09ab747442f08ea4 2023-04-12 02:55:52,468 DEBUG [StoreCloser-testSplitTableRegion,,1681268150430.9a391ed3cdb851dca0e4ca65b3a980e8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/9a391ed3cdb851dca0e4ca65b3a980e8/cf1/72636f8bfb49470784b49ea8615a5c6a.c426152f6c5cdf7d09ab747442f08ea4 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegion/9a391ed3cdb851dca0e4ca65b3a980e8/cf1/72636f8bfb49470784b49ea8615a5c6a.c426152f6c5cdf7d09ab747442f08ea4 2023-04-12 02:55:52,469 DEBUG [StoreCloser-testSplitTableRegion,,1681268150430.9a391ed3cdb851dca0e4ca65b3a980e8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/9a391ed3cdb851dca0e4ca65b3a980e8/cf1/3e014fee55b841d9a93f29d49aa55156.c426152f6c5cdf7d09ab747442f08ea4 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegion/9a391ed3cdb851dca0e4ca65b3a980e8/cf1/3e014fee55b841d9a93f29d49aa55156.c426152f6c5cdf7d09ab747442f08ea4 2023-04-12 02:55:52,470 DEBUG [StoreCloser-testSplitTableRegion,,1681268150430.9a391ed3cdb851dca0e4ca65b3a980e8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/9a391ed3cdb851dca0e4ca65b3a980e8/cf1/52c29c1aa57f4576b791c97686dd2e0d.c426152f6c5cdf7d09ab747442f08ea4 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegion/9a391ed3cdb851dca0e4ca65b3a980e8/cf1/52c29c1aa57f4576b791c97686dd2e0d.c426152f6c5cdf7d09ab747442f08ea4 2023-04-12 02:55:52,471 DEBUG [StoreCloser-testSplitTableRegion,,1681268150430.9a391ed3cdb851dca0e4ca65b3a980e8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/9a391ed3cdb851dca0e4ca65b3a980e8/cf1/a54d3a0a458d49dbbc18bbd89c01e473.c426152f6c5cdf7d09ab747442f08ea4 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegion/9a391ed3cdb851dca0e4ca65b3a980e8/cf1/a54d3a0a458d49dbbc18bbd89c01e473.c426152f6c5cdf7d09ab747442f08ea4 2023-04-12 02:55:52,471 DEBUG [StoreCloser-testSplitTableRegion,,1681268150430.9a391ed3cdb851dca0e4ca65b3a980e8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/9a391ed3cdb851dca0e4ca65b3a980e8/cf1/79de355eab9c46f98c089a49fcf5f846.c426152f6c5cdf7d09ab747442f08ea4 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegion/9a391ed3cdb851dca0e4ca65b3a980e8/cf1/79de355eab9c46f98c089a49fcf5f846.c426152f6c5cdf7d09ab747442f08ea4 2023-04-12 02:55:52,472 DEBUG [StoreCloser-testSplitTableRegion,,1681268150430.9a391ed3cdb851dca0e4ca65b3a980e8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/9a391ed3cdb851dca0e4ca65b3a980e8/cf1/1f290c1ae6ff4ac9aa79371af93b04cf.c426152f6c5cdf7d09ab747442f08ea4 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegion/9a391ed3cdb851dca0e4ca65b3a980e8/cf1/1f290c1ae6ff4ac9aa79371af93b04cf.c426152f6c5cdf7d09ab747442f08ea4 2023-04-12 02:55:52,473 DEBUG [StoreCloser-testSplitTableRegion,,1681268150430.9a391ed3cdb851dca0e4ca65b3a980e8.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/9a391ed3cdb851dca0e4ca65b3a980e8/cf2/ba426c6436064f9ca5ca16cecc0d8c98.c426152f6c5cdf7d09ab747442f08ea4->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/cf2/ba426c6436064f9ca5ca16cecc0d8c98-bottom, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/9a391ed3cdb851dca0e4ca65b3a980e8/cf2/f5ea9d0c0a4c491f91c40210e3a13892.c426152f6c5cdf7d09ab747442f08ea4->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/cf2/f5ea9d0c0a4c491f91c40210e3a13892-bottom, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/9a391ed3cdb851dca0e4ca65b3a980e8/cf2/c4a22b15c4714bc1a6f7b206d3315186.c426152f6c5cdf7d09ab747442f08ea4->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/cf2/c4a22b15c4714bc1a6f7b206d3315186-bottom, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/9a391ed3cdb851dca0e4ca65b3a980e8/cf2/e4c7205c2e024488b0db4154888f3f50.c426152f6c5cdf7d09ab747442f08ea4->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/cf2/e4c7205c2e024488b0db4154888f3f50-bottom, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/9a391ed3cdb851dca0e4ca65b3a980e8/cf2/e7cb094353e04c1c89a6b020574e3da1.c426152f6c5cdf7d09ab747442f08ea4->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/cf2/e7cb094353e04c1c89a6b020574e3da1-bottom, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/9a391ed3cdb851dca0e4ca65b3a980e8/cf2/506165cd013640888b0ebd392bd533ab.c426152f6c5cdf7d09ab747442f08ea4->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/cf2/506165cd013640888b0ebd392bd533ab-bottom, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/9a391ed3cdb851dca0e4ca65b3a980e8/cf2/5baff79a42ff479888e579a42af26748.c426152f6c5cdf7d09ab747442f08ea4->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/cf2/5baff79a42ff479888e579a42af26748-bottom] to archive 2023-04-12 02:55:52,474 DEBUG [StoreCloser-testSplitTableRegion,,1681268150430.9a391ed3cdb851dca0e4ca65b3a980e8.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-12 02:55:52,475 DEBUG [StoreCloser-testSplitTableRegion,,1681268150430.9a391ed3cdb851dca0e4ca65b3a980e8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/9a391ed3cdb851dca0e4ca65b3a980e8/cf2/ba426c6436064f9ca5ca16cecc0d8c98.c426152f6c5cdf7d09ab747442f08ea4 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegion/9a391ed3cdb851dca0e4ca65b3a980e8/cf2/ba426c6436064f9ca5ca16cecc0d8c98.c426152f6c5cdf7d09ab747442f08ea4 2023-04-12 02:55:52,476 DEBUG [StoreCloser-testSplitTableRegion,,1681268150430.9a391ed3cdb851dca0e4ca65b3a980e8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/9a391ed3cdb851dca0e4ca65b3a980e8/cf2/f5ea9d0c0a4c491f91c40210e3a13892.c426152f6c5cdf7d09ab747442f08ea4 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegion/9a391ed3cdb851dca0e4ca65b3a980e8/cf2/f5ea9d0c0a4c491f91c40210e3a13892.c426152f6c5cdf7d09ab747442f08ea4 2023-04-12 02:55:52,477 DEBUG [StoreCloser-testSplitTableRegion,,1681268150430.9a391ed3cdb851dca0e4ca65b3a980e8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/9a391ed3cdb851dca0e4ca65b3a980e8/cf2/c4a22b15c4714bc1a6f7b206d3315186.c426152f6c5cdf7d09ab747442f08ea4 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegion/9a391ed3cdb851dca0e4ca65b3a980e8/cf2/c4a22b15c4714bc1a6f7b206d3315186.c426152f6c5cdf7d09ab747442f08ea4 2023-04-12 02:55:52,478 DEBUG [StoreCloser-testSplitTableRegion,,1681268150430.9a391ed3cdb851dca0e4ca65b3a980e8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/9a391ed3cdb851dca0e4ca65b3a980e8/cf2/e4c7205c2e024488b0db4154888f3f50.c426152f6c5cdf7d09ab747442f08ea4 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegion/9a391ed3cdb851dca0e4ca65b3a980e8/cf2/e4c7205c2e024488b0db4154888f3f50.c426152f6c5cdf7d09ab747442f08ea4 2023-04-12 02:55:52,479 DEBUG [StoreCloser-testSplitTableRegion,,1681268150430.9a391ed3cdb851dca0e4ca65b3a980e8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/9a391ed3cdb851dca0e4ca65b3a980e8/cf2/e7cb094353e04c1c89a6b020574e3da1.c426152f6c5cdf7d09ab747442f08ea4 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegion/9a391ed3cdb851dca0e4ca65b3a980e8/cf2/e7cb094353e04c1c89a6b020574e3da1.c426152f6c5cdf7d09ab747442f08ea4 2023-04-12 02:55:52,480 DEBUG [StoreCloser-testSplitTableRegion,,1681268150430.9a391ed3cdb851dca0e4ca65b3a980e8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/9a391ed3cdb851dca0e4ca65b3a980e8/cf2/506165cd013640888b0ebd392bd533ab.c426152f6c5cdf7d09ab747442f08ea4 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegion/9a391ed3cdb851dca0e4ca65b3a980e8/cf2/506165cd013640888b0ebd392bd533ab.c426152f6c5cdf7d09ab747442f08ea4 2023-04-12 02:55:52,480 DEBUG [StoreCloser-testSplitTableRegion,,1681268150430.9a391ed3cdb851dca0e4ca65b3a980e8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/9a391ed3cdb851dca0e4ca65b3a980e8/cf2/5baff79a42ff479888e579a42af26748.c426152f6c5cdf7d09ab747442f08ea4 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegion/9a391ed3cdb851dca0e4ca65b3a980e8/cf2/5baff79a42ff479888e579a42af26748.c426152f6c5cdf7d09ab747442f08ea4 2023-04-12 02:55:52,483 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/9a391ed3cdb851dca0e4ca65b3a980e8/recovered.edits/90.seqid, newMaxSeqId=90, maxSeqId=85 2023-04-12 02:55:52,484 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro 2023-04-12 02:55:52,484 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed testSplitTableRegion,,1681268150430.9a391ed3cdb851dca0e4ca65b3a980e8. 2023-04-12 02:55:52,484 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 9a391ed3cdb851dca0e4ca65b3a980e8: 2023-04-12 02:55:52,485 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 9a391ed3cdb851dca0e4ca65b3a980e8 2023-04-12 02:55:52,485 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close c9ca1dc933715e878706f11781755586 2023-04-12 02:55:52,485 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing c9ca1dc933715e878706f11781755586, disabling compactions & flushes 2023-04-12 02:55:52,485 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region testSplitTableRegion,41,1681268150430.c9ca1dc933715e878706f11781755586. 2023-04-12 02:55:52,485 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testSplitTableRegion,41,1681268150430.c9ca1dc933715e878706f11781755586. 2023-04-12 02:55:52,485 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on testSplitTableRegion,41,1681268150430.c9ca1dc933715e878706f11781755586. after waiting 0 ms 2023-04-12 02:55:52,485 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region testSplitTableRegion,41,1681268150430.c9ca1dc933715e878706f11781755586. 2023-04-12 02:55:52,485 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=152 updating hbase:meta row=9a391ed3cdb851dca0e4ca65b3a980e8, regionState=CLOSED 2023-04-12 02:55:52,485 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testSplitTableRegion,,1681268150430.9a391ed3cdb851dca0e4ca65b3a980e8.","families":{"info":[{"qualifier":"regioninfo","vlen":56,"tag":[],"timestamp":"1681268152485"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681268152485"}]},"ts":"1681268152485"} 2023-04-12 02:55:52,486 DEBUG [StoreCloser-testSplitTableRegion,41,1681268150430.c9ca1dc933715e878706f11781755586.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c9ca1dc933715e878706f11781755586/cf1/bd8f39350eae4e18bebd947afb5d9f5e.c426152f6c5cdf7d09ab747442f08ea4->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/cf1/bd8f39350eae4e18bebd947afb5d9f5e-top, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c9ca1dc933715e878706f11781755586/cf1/72636f8bfb49470784b49ea8615a5c6a.c426152f6c5cdf7d09ab747442f08ea4->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/cf1/72636f8bfb49470784b49ea8615a5c6a-top, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c9ca1dc933715e878706f11781755586/cf1/3e014fee55b841d9a93f29d49aa55156.c426152f6c5cdf7d09ab747442f08ea4->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/cf1/3e014fee55b841d9a93f29d49aa55156-top, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c9ca1dc933715e878706f11781755586/cf1/52c29c1aa57f4576b791c97686dd2e0d.c426152f6c5cdf7d09ab747442f08ea4->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/cf1/52c29c1aa57f4576b791c97686dd2e0d-top, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c9ca1dc933715e878706f11781755586/cf1/a54d3a0a458d49dbbc18bbd89c01e473.c426152f6c5cdf7d09ab747442f08ea4->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/cf1/a54d3a0a458d49dbbc18bbd89c01e473-top, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c9ca1dc933715e878706f11781755586/cf1/79de355eab9c46f98c089a49fcf5f846.c426152f6c5cdf7d09ab747442f08ea4->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/cf1/79de355eab9c46f98c089a49fcf5f846-top, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c9ca1dc933715e878706f11781755586/cf1/1f290c1ae6ff4ac9aa79371af93b04cf.c426152f6c5cdf7d09ab747442f08ea4->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/cf1/1f290c1ae6ff4ac9aa79371af93b04cf-top] to archive 2023-04-12 02:55:52,487 DEBUG [StoreCloser-testSplitTableRegion,41,1681268150430.c9ca1dc933715e878706f11781755586.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-12 02:55:52,488 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=155, resume processing ppid=152 2023-04-12 02:55:52,488 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=155, ppid=152, state=SUCCESS; CloseRegionProcedure 9a391ed3cdb851dca0e4ca65b3a980e8, server=jenkins-hbase4.apache.org,34579,1681268031719 in 171 msec 2023-04-12 02:55:52,488 DEBUG [StoreCloser-testSplitTableRegion,41,1681268150430.c9ca1dc933715e878706f11781755586.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c9ca1dc933715e878706f11781755586/cf1/bd8f39350eae4e18bebd947afb5d9f5e.c426152f6c5cdf7d09ab747442f08ea4 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegion/c9ca1dc933715e878706f11781755586/cf1/bd8f39350eae4e18bebd947afb5d9f5e.c426152f6c5cdf7d09ab747442f08ea4 2023-04-12 02:55:52,489 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=152, ppid=151, state=SUCCESS; TransitRegionStateProcedure table=testSplitTableRegion, region=9a391ed3cdb851dca0e4ca65b3a980e8, UNASSIGN in 177 msec 2023-04-12 02:55:52,489 DEBUG [StoreCloser-testSplitTableRegion,41,1681268150430.c9ca1dc933715e878706f11781755586.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c9ca1dc933715e878706f11781755586/cf1/72636f8bfb49470784b49ea8615a5c6a.c426152f6c5cdf7d09ab747442f08ea4 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegion/c9ca1dc933715e878706f11781755586/cf1/72636f8bfb49470784b49ea8615a5c6a.c426152f6c5cdf7d09ab747442f08ea4 2023-04-12 02:55:52,490 DEBUG [StoreCloser-testSplitTableRegion,41,1681268150430.c9ca1dc933715e878706f11781755586.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c9ca1dc933715e878706f11781755586/cf1/3e014fee55b841d9a93f29d49aa55156.c426152f6c5cdf7d09ab747442f08ea4 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegion/c9ca1dc933715e878706f11781755586/cf1/3e014fee55b841d9a93f29d49aa55156.c426152f6c5cdf7d09ab747442f08ea4 2023-04-12 02:55:52,490 DEBUG [StoreCloser-testSplitTableRegion,41,1681268150430.c9ca1dc933715e878706f11781755586.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c9ca1dc933715e878706f11781755586/cf1/52c29c1aa57f4576b791c97686dd2e0d.c426152f6c5cdf7d09ab747442f08ea4 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegion/c9ca1dc933715e878706f11781755586/cf1/52c29c1aa57f4576b791c97686dd2e0d.c426152f6c5cdf7d09ab747442f08ea4 2023-04-12 02:55:52,491 DEBUG [StoreCloser-testSplitTableRegion,41,1681268150430.c9ca1dc933715e878706f11781755586.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c9ca1dc933715e878706f11781755586/cf1/a54d3a0a458d49dbbc18bbd89c01e473.c426152f6c5cdf7d09ab747442f08ea4 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegion/c9ca1dc933715e878706f11781755586/cf1/a54d3a0a458d49dbbc18bbd89c01e473.c426152f6c5cdf7d09ab747442f08ea4 2023-04-12 02:55:52,492 DEBUG [StoreCloser-testSplitTableRegion,41,1681268150430.c9ca1dc933715e878706f11781755586.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c9ca1dc933715e878706f11781755586/cf1/79de355eab9c46f98c089a49fcf5f846.c426152f6c5cdf7d09ab747442f08ea4 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegion/c9ca1dc933715e878706f11781755586/cf1/79de355eab9c46f98c089a49fcf5f846.c426152f6c5cdf7d09ab747442f08ea4 2023-04-12 02:55:52,493 DEBUG [StoreCloser-testSplitTableRegion,41,1681268150430.c9ca1dc933715e878706f11781755586.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c9ca1dc933715e878706f11781755586/cf1/1f290c1ae6ff4ac9aa79371af93b04cf.c426152f6c5cdf7d09ab747442f08ea4 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegion/c9ca1dc933715e878706f11781755586/cf1/1f290c1ae6ff4ac9aa79371af93b04cf.c426152f6c5cdf7d09ab747442f08ea4 2023-04-12 02:55:52,494 DEBUG [StoreCloser-testSplitTableRegion,41,1681268150430.c9ca1dc933715e878706f11781755586.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c9ca1dc933715e878706f11781755586/cf2/ba426c6436064f9ca5ca16cecc0d8c98.c426152f6c5cdf7d09ab747442f08ea4->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/cf2/ba426c6436064f9ca5ca16cecc0d8c98-top, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c9ca1dc933715e878706f11781755586/cf2/f5ea9d0c0a4c491f91c40210e3a13892.c426152f6c5cdf7d09ab747442f08ea4->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/cf2/f5ea9d0c0a4c491f91c40210e3a13892-top, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c9ca1dc933715e878706f11781755586/cf2/c4a22b15c4714bc1a6f7b206d3315186.c426152f6c5cdf7d09ab747442f08ea4->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/cf2/c4a22b15c4714bc1a6f7b206d3315186-top, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c9ca1dc933715e878706f11781755586/cf2/e4c7205c2e024488b0db4154888f3f50.c426152f6c5cdf7d09ab747442f08ea4->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/cf2/e4c7205c2e024488b0db4154888f3f50-top, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c9ca1dc933715e878706f11781755586/cf2/e7cb094353e04c1c89a6b020574e3da1.c426152f6c5cdf7d09ab747442f08ea4->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/cf2/e7cb094353e04c1c89a6b020574e3da1-top, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c9ca1dc933715e878706f11781755586/cf2/506165cd013640888b0ebd392bd533ab.c426152f6c5cdf7d09ab747442f08ea4->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/cf2/506165cd013640888b0ebd392bd533ab-top, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c9ca1dc933715e878706f11781755586/cf2/5baff79a42ff479888e579a42af26748.c426152f6c5cdf7d09ab747442f08ea4->hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/cf2/5baff79a42ff479888e579a42af26748-top] to archive 2023-04-12 02:55:52,495 DEBUG [StoreCloser-testSplitTableRegion,41,1681268150430.c9ca1dc933715e878706f11781755586.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-12 02:55:52,496 DEBUG [StoreCloser-testSplitTableRegion,41,1681268150430.c9ca1dc933715e878706f11781755586.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c9ca1dc933715e878706f11781755586/cf2/ba426c6436064f9ca5ca16cecc0d8c98.c426152f6c5cdf7d09ab747442f08ea4 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegion/c9ca1dc933715e878706f11781755586/cf2/ba426c6436064f9ca5ca16cecc0d8c98.c426152f6c5cdf7d09ab747442f08ea4 2023-04-12 02:55:52,496 DEBUG [StoreCloser-testSplitTableRegion,41,1681268150430.c9ca1dc933715e878706f11781755586.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c9ca1dc933715e878706f11781755586/cf2/f5ea9d0c0a4c491f91c40210e3a13892.c426152f6c5cdf7d09ab747442f08ea4 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegion/c9ca1dc933715e878706f11781755586/cf2/f5ea9d0c0a4c491f91c40210e3a13892.c426152f6c5cdf7d09ab747442f08ea4 2023-04-12 02:55:52,497 DEBUG [StoreCloser-testSplitTableRegion,41,1681268150430.c9ca1dc933715e878706f11781755586.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c9ca1dc933715e878706f11781755586/cf2/c4a22b15c4714bc1a6f7b206d3315186.c426152f6c5cdf7d09ab747442f08ea4 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegion/c9ca1dc933715e878706f11781755586/cf2/c4a22b15c4714bc1a6f7b206d3315186.c426152f6c5cdf7d09ab747442f08ea4 2023-04-12 02:55:52,498 DEBUG [StoreCloser-testSplitTableRegion,41,1681268150430.c9ca1dc933715e878706f11781755586.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c9ca1dc933715e878706f11781755586/cf2/e4c7205c2e024488b0db4154888f3f50.c426152f6c5cdf7d09ab747442f08ea4 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegion/c9ca1dc933715e878706f11781755586/cf2/e4c7205c2e024488b0db4154888f3f50.c426152f6c5cdf7d09ab747442f08ea4 2023-04-12 02:55:52,498 DEBUG [StoreCloser-testSplitTableRegion,41,1681268150430.c9ca1dc933715e878706f11781755586.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c9ca1dc933715e878706f11781755586/cf2/e7cb094353e04c1c89a6b020574e3da1.c426152f6c5cdf7d09ab747442f08ea4 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegion/c9ca1dc933715e878706f11781755586/cf2/e7cb094353e04c1c89a6b020574e3da1.c426152f6c5cdf7d09ab747442f08ea4 2023-04-12 02:55:52,499 DEBUG [StoreCloser-testSplitTableRegion,41,1681268150430.c9ca1dc933715e878706f11781755586.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c9ca1dc933715e878706f11781755586/cf2/506165cd013640888b0ebd392bd533ab.c426152f6c5cdf7d09ab747442f08ea4 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegion/c9ca1dc933715e878706f11781755586/cf2/506165cd013640888b0ebd392bd533ab.c426152f6c5cdf7d09ab747442f08ea4 2023-04-12 02:55:52,500 DEBUG [StoreCloser-testSplitTableRegion,41,1681268150430.c9ca1dc933715e878706f11781755586.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c9ca1dc933715e878706f11781755586/cf2/5baff79a42ff479888e579a42af26748.c426152f6c5cdf7d09ab747442f08ea4 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegion/c9ca1dc933715e878706f11781755586/cf2/5baff79a42ff479888e579a42af26748.c426152f6c5cdf7d09ab747442f08ea4 2023-04-12 02:55:52,502 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/default/testSplitTableRegion/c9ca1dc933715e878706f11781755586/recovered.edits/90.seqid, newMaxSeqId=90, maxSeqId=85 2023-04-12 02:55:52,503 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro 2023-04-12 02:55:52,503 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed testSplitTableRegion,41,1681268150430.c9ca1dc933715e878706f11781755586. 2023-04-12 02:55:52,503 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for c9ca1dc933715e878706f11781755586: 2023-04-12 02:55:52,504 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed c9ca1dc933715e878706f11781755586 2023-04-12 02:55:52,504 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=153 updating hbase:meta row=c9ca1dc933715e878706f11781755586, regionState=CLOSED 2023-04-12 02:55:52,504 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testSplitTableRegion,41,1681268150430.c9ca1dc933715e878706f11781755586.","families":{"info":[{"qualifier":"regioninfo","vlen":56,"tag":[],"timestamp":"1681268152504"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1681268152504"}]},"ts":"1681268152504"} 2023-04-12 02:55:52,506 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=154, resume processing ppid=153 2023-04-12 02:55:52,506 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=154, ppid=153, state=SUCCESS; CloseRegionProcedure c9ca1dc933715e878706f11781755586, server=jenkins-hbase4.apache.org,34579,1681268031719 in 192 msec 2023-04-12 02:55:52,507 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=153, resume processing ppid=151 2023-04-12 02:55:52,507 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=153, ppid=151, state=SUCCESS; TransitRegionStateProcedure table=testSplitTableRegion, region=c9ca1dc933715e878706f11781755586, UNASSIGN in 195 msec 2023-04-12 02:55:52,508 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testSplitTableRegion","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1681268152508"}]},"ts":"1681268152508"} 2023-04-12 02:55:52,508 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testSplitTableRegion, state=DISABLED in hbase:meta 2023-04-12 02:55:52,510 INFO [PEWorker-1] procedure.DisableTableProcedure(305): Set testSplitTableRegion to state=DISABLED 2023-04-12 02:55:52,511 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=151, state=SUCCESS; DisableTableProcedure table=testSplitTableRegion in 205 msec 2023-04-12 02:55:52,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] master.MasterRpcServices(1227): Checking to see if procedure is done pid=151 2023-04-12 02:55:52,609 INFO [Listener at localhost/46667] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testSplitTableRegion, procId: 151 completed 2023-04-12 02:55:52,610 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete testSplitTableRegion 2023-04-12 02:55:52,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] procedure2.ProcedureExecutor(1029): Stored pid=156, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=testSplitTableRegion 2023-04-12 02:55:52,612 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=156, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=testSplitTableRegion 2023-04-12 02:55:52,613 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=156, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=testSplitTableRegion 2023-04-12 02:55:52,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] master.MasterRpcServices(1227): Checking to see if procedure is done pid=156 2023-04-12 02:55:52,616 DEBUG [HFileArchiver-13] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4 2023-04-12 02:55:52,616 DEBUG [HFileArchiver-15] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitTableRegion/c9ca1dc933715e878706f11781755586 2023-04-12 02:55:52,616 DEBUG [HFileArchiver-14] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitTableRegion/9a391ed3cdb851dca0e4ca65b3a980e8 2023-04-12 02:55:52,618 DEBUG [HFileArchiver-13] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/cf1, FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/cf2, FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/recovered.edits] 2023-04-12 02:55:52,618 DEBUG [HFileArchiver-14] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitTableRegion/9a391ed3cdb851dca0e4ca65b3a980e8/cf1, FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitTableRegion/9a391ed3cdb851dca0e4ca65b3a980e8/cf2, FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitTableRegion/9a391ed3cdb851dca0e4ca65b3a980e8/recovered.edits] 2023-04-12 02:55:52,618 DEBUG [HFileArchiver-15] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitTableRegion/c9ca1dc933715e878706f11781755586/cf1, FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitTableRegion/c9ca1dc933715e878706f11781755586/cf2, FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitTableRegion/c9ca1dc933715e878706f11781755586/recovered.edits] 2023-04-12 02:55:52,620 DEBUG [HFileArchiver-14] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitTableRegion/9a391ed3cdb851dca0e4ca65b3a980e8/cf1/815b43de660541e097ab58c3b0dcf022 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegion/9a391ed3cdb851dca0e4ca65b3a980e8/cf1/815b43de660541e097ab58c3b0dcf022 2023-04-12 02:55:52,620 DEBUG [HFileArchiver-15] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitTableRegion/c9ca1dc933715e878706f11781755586/cf1/3199801c87964457a97c9936a9f2221d to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegion/c9ca1dc933715e878706f11781755586/cf1/3199801c87964457a97c9936a9f2221d 2023-04-12 02:55:52,621 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/cf1/1f290c1ae6ff4ac9aa79371af93b04cf to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/cf1/1f290c1ae6ff4ac9aa79371af93b04cf 2023-04-12 02:55:52,622 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/cf1/3e014fee55b841d9a93f29d49aa55156 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/cf1/3e014fee55b841d9a93f29d49aa55156 2023-04-12 02:55:52,622 DEBUG [HFileArchiver-14] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitTableRegion/9a391ed3cdb851dca0e4ca65b3a980e8/cf2/f78639c04e9f489c9d0735c8ce312492 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegion/9a391ed3cdb851dca0e4ca65b3a980e8/cf2/f78639c04e9f489c9d0735c8ce312492 2023-04-12 02:55:52,623 DEBUG [HFileArchiver-15] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitTableRegion/c9ca1dc933715e878706f11781755586/cf2/7745c9273e724bbb91b3a05cbeb0bf13 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegion/c9ca1dc933715e878706f11781755586/cf2/7745c9273e724bbb91b3a05cbeb0bf13 2023-04-12 02:55:52,623 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/cf1/52c29c1aa57f4576b791c97686dd2e0d to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/cf1/52c29c1aa57f4576b791c97686dd2e0d 2023-04-12 02:55:52,627 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/cf1/72636f8bfb49470784b49ea8615a5c6a to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/cf1/72636f8bfb49470784b49ea8615a5c6a 2023-04-12 02:55:52,628 DEBUG [HFileArchiver-14] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitTableRegion/9a391ed3cdb851dca0e4ca65b3a980e8/recovered.edits/90.seqid to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegion/9a391ed3cdb851dca0e4ca65b3a980e8/recovered.edits/90.seqid 2023-04-12 02:55:52,628 DEBUG [HFileArchiver-15] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitTableRegion/c9ca1dc933715e878706f11781755586/recovered.edits/90.seqid to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegion/c9ca1dc933715e878706f11781755586/recovered.edits/90.seqid 2023-04-12 02:55:52,628 DEBUG [HFileArchiver-14] backup.HFileArchiver(596): Deleted hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitTableRegion/9a391ed3cdb851dca0e4ca65b3a980e8 2023-04-12 02:55:52,629 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/cf1/79de355eab9c46f98c089a49fcf5f846 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/cf1/79de355eab9c46f98c089a49fcf5f846 2023-04-12 02:55:52,629 DEBUG [HFileArchiver-15] backup.HFileArchiver(596): Deleted hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitTableRegion/c9ca1dc933715e878706f11781755586 2023-04-12 02:55:52,630 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/cf1/a54d3a0a458d49dbbc18bbd89c01e473 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/cf1/a54d3a0a458d49dbbc18bbd89c01e473 2023-04-12 02:55:52,631 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/cf1/bd8f39350eae4e18bebd947afb5d9f5e to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/cf1/bd8f39350eae4e18bebd947afb5d9f5e 2023-04-12 02:55:52,633 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/cf2/506165cd013640888b0ebd392bd533ab to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/cf2/506165cd013640888b0ebd392bd533ab 2023-04-12 02:55:52,634 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/cf2/5baff79a42ff479888e579a42af26748 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/cf2/5baff79a42ff479888e579a42af26748 2023-04-12 02:55:52,635 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/cf2/ba426c6436064f9ca5ca16cecc0d8c98 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/cf2/ba426c6436064f9ca5ca16cecc0d8c98 2023-04-12 02:55:52,636 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/cf2/c4a22b15c4714bc1a6f7b206d3315186 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/cf2/c4a22b15c4714bc1a6f7b206d3315186 2023-04-12 02:55:52,637 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/cf2/e4c7205c2e024488b0db4154888f3f50 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/cf2/e4c7205c2e024488b0db4154888f3f50 2023-04-12 02:55:52,637 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/cf2/e7cb094353e04c1c89a6b020574e3da1 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/cf2/e7cb094353e04c1c89a6b020574e3da1 2023-04-12 02:55:52,638 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/cf2/f5ea9d0c0a4c491f91c40210e3a13892 to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/cf2/f5ea9d0c0a4c491f91c40210e3a13892 2023-04-12 02:55:52,640 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/recovered.edits/85.seqid to hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/archive/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4/recovered.edits/85.seqid 2023-04-12 02:55:52,640 DEBUG [HFileArchiver-13] backup.HFileArchiver(596): Deleted hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/.tmp/data/default/testSplitTableRegion/c426152f6c5cdf7d09ab747442f08ea4 2023-04-12 02:55:52,640 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived testSplitTableRegion regions 2023-04-12 02:55:52,642 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=156, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=testSplitTableRegion 2023-04-12 02:55:52,643 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 3 rows of testSplitTableRegion from hbase:meta 2023-04-12 02:55:52,645 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'testSplitTableRegion' descriptor. 2023-04-12 02:55:52,645 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=156, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=testSplitTableRegion 2023-04-12 02:55:52,646 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'testSplitTableRegion' from region states. 2023-04-12 02:55:52,646 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testSplitTableRegion,,1681268148351.c426152f6c5cdf7d09ab747442f08ea4.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1681268152646"}]},"ts":"9223372036854775807"} 2023-04-12 02:55:52,646 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testSplitTableRegion,,1681268150430.9a391ed3cdb851dca0e4ca65b3a980e8.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1681268152646"}]},"ts":"9223372036854775807"} 2023-04-12 02:55:52,646 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testSplitTableRegion,41,1681268150430.c9ca1dc933715e878706f11781755586.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1681268152646"}]},"ts":"9223372036854775807"} 2023-04-12 02:55:52,647 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 3 regions from META 2023-04-12 02:55:52,647 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => c426152f6c5cdf7d09ab747442f08ea4, NAME => 'testSplitTableRegion,,1681268148351.c426152f6c5cdf7d09ab747442f08ea4.', STARTKEY => '', ENDKEY => ''}, {ENCODED => 9a391ed3cdb851dca0e4ca65b3a980e8, NAME => 'testSplitTableRegion,,1681268150430.9a391ed3cdb851dca0e4ca65b3a980e8.', STARTKEY => '', ENDKEY => '41'}, {ENCODED => c9ca1dc933715e878706f11781755586, NAME => 'testSplitTableRegion,41,1681268150430.c9ca1dc933715e878706f11781755586.', STARTKEY => '41', ENDKEY => ''}] 2023-04-12 02:55:52,647 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'testSplitTableRegion' as deleted. 2023-04-12 02:55:52,647 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testSplitTableRegion","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1681268152647"}]},"ts":"9223372036854775807"} 2023-04-12 02:55:52,648 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table testSplitTableRegion state from META 2023-04-12 02:55:52,650 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=156, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=testSplitTableRegion 2023-04-12 02:55:52,651 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=156, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegion in 40 msec 2023-04-12 02:55:52,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=40281] master.MasterRpcServices(1227): Checking to see if procedure is done pid=156 2023-04-12 02:55:52,714 INFO [Listener at localhost/46667] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:testSplitTableRegion, procId: 156 completed 2023-04-12 02:55:52,728 INFO [Listener at localhost/46667] hbase.ResourceChecker(175): after: master.assignment.TestSplitTableRegionProcedure#testSplitTableRegion Thread=376 (was 373) - Thread LEAK? -, OpenFileDescriptor=830 (was 819) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=333 (was 362), ProcessCount=170 (was 170), AvailableMemoryMB=758 (was 767) 2023-04-12 02:55:52,728 INFO [Listener at localhost/46667] hbase.HBaseTestingUtility(1286): Shutting down minicluster 2023-04-12 02:55:52,728 INFO [Listener at localhost/46667] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-12 02:55:52,729 DEBUG [Listener at localhost/46667] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7b7d11f1 to 127.0.0.1:57543 2023-04-12 02:55:52,729 DEBUG [Listener at localhost/46667] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-12 02:55:52,729 DEBUG [Listener at localhost/46667] util.JVMClusterUtil(237): Shutting down HBase Cluster 2023-04-12 02:55:52,729 DEBUG [Listener at localhost/46667] util.JVMClusterUtil(257): Found active master hash=1958884826, stopped=false 2023-04-12 02:55:52,730 INFO [Listener at localhost/46667] master.ServerManager(901): Cluster shutdown requested of master=jenkins-hbase4.apache.org,40281,1681268030476 2023-04-12 02:55:52,732 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-04-12 02:55:52,732 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-04-12 02:55:52,732 INFO [Listener at localhost/46667] procedure2.ProcedureExecutor(629): Stopping 2023-04-12 02:55:52,732 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-04-12 02:55:52,732 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-12 02:55:52,732 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-04-12 02:55:52,732 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-12 02:55:52,732 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-12 02:55:52,732 DEBUG [Listener at localhost/46667] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1295f4ce to 127.0.0.1:57543 2023-04-12 02:55:52,733 DEBUG [Listener at localhost/46667] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-12 02:55:52,733 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-12 02:55:52,733 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-12 02:55:52,733 INFO [Listener at localhost/46667] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,34579,1681268031719' ***** 2023-04-12 02:55:52,733 INFO [Listener at localhost/46667] regionserver.HRegionServer(2310): STOPPED: Shutdown requested 2023-04-12 02:55:52,733 INFO [Listener at localhost/46667] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,44679,1681268031772' ***** 2023-04-12 02:55:52,733 INFO [Listener at localhost/46667] regionserver.HRegionServer(2310): STOPPED: Shutdown requested 2023-04-12 02:55:52,733 INFO [Listener at localhost/46667] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,43091,1681268031804' ***** 2023-04-12 02:55:52,733 INFO [Listener at localhost/46667] regionserver.HRegionServer(2310): STOPPED: Shutdown requested 2023-04-12 02:55:52,733 INFO [RS:1;jenkins-hbase4:44679] regionserver.HeapMemoryManager(220): Stopping 2023-04-12 02:55:52,733 INFO [RS:1;jenkins-hbase4:44679] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-04-12 02:55:52,734 INFO [RS:0;jenkins-hbase4:34579] regionserver.HeapMemoryManager(220): Stopping 2023-04-12 02:55:52,734 INFO [RS:1;jenkins-hbase4:44679] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-04-12 02:55:52,734 INFO [RS:0;jenkins-hbase4:34579] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-04-12 02:55:52,734 INFO [RS:0;jenkins-hbase4:34579] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-04-12 02:55:52,734 INFO [RS:0;jenkins-hbase4:34579] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:52,734 INFO [RS:1;jenkins-hbase4:44679] regionserver.HRegionServer(3304): Received CLOSE for 1f255e980a31bb266f2206a30aa2ae3f 2023-04-12 02:55:52,734 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-04-12 02:55:52,734 INFO [RS:1;jenkins-hbase4:44679] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:52,733 INFO [RS:2;jenkins-hbase4:43091] regionserver.HeapMemoryManager(220): Stopping 2023-04-12 02:55:52,735 DEBUG [RS:1;jenkins-hbase4:44679] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x73adb104 to 127.0.0.1:57543 2023-04-12 02:55:52,735 INFO [RS:2;jenkins-hbase4:43091] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-04-12 02:55:52,734 DEBUG [RS:0;jenkins-hbase4:34579] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1d7fbc23 to 127.0.0.1:57543 2023-04-12 02:55:52,733 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-04-12 02:55:52,735 DEBUG [RS:0;jenkins-hbase4:34579] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-12 02:55:52,735 INFO [RS:2;jenkins-hbase4:43091] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-04-12 02:55:52,735 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-04-12 02:55:52,735 INFO [RS:2;jenkins-hbase4:43091] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:52,735 DEBUG [RS:1;jenkins-hbase4:44679] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-12 02:55:52,735 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 1f255e980a31bb266f2206a30aa2ae3f, disabling compactions & flushes 2023-04-12 02:55:52,735 INFO [RS:1;jenkins-hbase4:44679] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-04-12 02:55:52,735 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1681268033852.1f255e980a31bb266f2206a30aa2ae3f. 2023-04-12 02:55:52,735 INFO [RS:1;jenkins-hbase4:44679] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-04-12 02:55:52,735 INFO [RS:1;jenkins-hbase4:44679] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-04-12 02:55:52,735 DEBUG [RS:2;jenkins-hbase4:43091] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2c37bd45 to 127.0.0.1:57543 2023-04-12 02:55:52,735 INFO [RS:0;jenkins-hbase4:34579] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,34579,1681268031719; all regions closed. 2023-04-12 02:55:52,735 DEBUG [RS:2;jenkins-hbase4:43091] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-12 02:55:52,735 INFO [RS:1;jenkins-hbase4:44679] regionserver.HRegionServer(3304): Received CLOSE for 1588230740 2023-04-12 02:55:52,735 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1681268033852.1f255e980a31bb266f2206a30aa2ae3f. 2023-04-12 02:55:52,735 INFO [RS:2;jenkins-hbase4:43091] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,43091,1681268031804; all regions closed. 2023-04-12 02:55:52,736 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1681268033852.1f255e980a31bb266f2206a30aa2ae3f. after waiting 0 ms 2023-04-12 02:55:52,736 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1681268033852.1f255e980a31bb266f2206a30aa2ae3f. 2023-04-12 02:55:52,736 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 1f255e980a31bb266f2206a30aa2ae3f 1/1 column families, dataSize=78 B heapSize=488 B 2023-04-12 02:55:52,736 INFO [RS:1;jenkins-hbase4:44679] regionserver.HRegionServer(1475): Waiting on 2 regions to close 2023-04-12 02:55:52,736 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-04-12 02:55:52,736 DEBUG [RS:1;jenkins-hbase4:44679] regionserver.HRegionServer(1479): Online Regions={1f255e980a31bb266f2206a30aa2ae3f=hbase:namespace,,1681268033852.1f255e980a31bb266f2206a30aa2ae3f., 1588230740=hbase:meta,,1.1588230740} 2023-04-12 02:55:52,736 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-04-12 02:55:52,736 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-04-12 02:55:52,736 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-04-12 02:55:52,736 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-04-12 02:55:52,736 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 1588230740 3/3 column families, dataSize=79.40 KB heapSize=125.77 KB 2023-04-12 02:55:52,737 DEBUG [RS:1;jenkins-hbase4:44679] regionserver.HRegionServer(1505): Waiting on 1588230740, 1f255e980a31bb266f2206a30aa2ae3f 2023-04-12 02:55:52,892 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-04-12 02:55:52,892 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-04-12 02:55:52,892 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-04-12 02:55:52,894 INFO [AsyncFSWAL-0-hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f-prefix:jenkins-hbase4.apache.org,44679,1681268031772.meta] wal.AbstractFSWAL(1141): Slow sync cost: 155 ms, current pipeline: [DatanodeInfoWithStorage[127.0.0.1:37795,DS-420b97ca-a460-40e3-b9fb-b2ada02a0355,DISK], DatanodeInfoWithStorage[127.0.0.1:42215,DS-737d1c12-0cb5-4776-9e46-0ca9b86b7577,DISK], DatanodeInfoWithStorage[127.0.0.1:33739,DS-cdcda459-f3ea-4856-a051-b40e48414f68,DISK]] 2023-04-12 02:55:52,904 DEBUG [RS:2;jenkins-hbase4:43091] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/oldWALs 2023-04-12 02:55:52,904 INFO [RS:2;jenkins-hbase4:43091] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C43091%2C1681268031804:(num 1681268033369) 2023-04-12 02:55:52,904 DEBUG [RS:2;jenkins-hbase4:43091] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-12 02:55:52,904 INFO [RS:2;jenkins-hbase4:43091] regionserver.LeaseManager(133): Closed leases 2023-04-12 02:55:52,905 DEBUG [RS:0;jenkins-hbase4:34579] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/oldWALs 2023-04-12 02:55:52,905 INFO [RS:0;jenkins-hbase4:34579] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C34579%2C1681268031719:(num 1681268033370) 2023-04-12 02:55:52,905 DEBUG [RS:0;jenkins-hbase4:34579] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-12 02:55:52,905 INFO [RS:0;jenkins-hbase4:34579] regionserver.LeaseManager(133): Closed leases 2023-04-12 02:55:52,906 INFO [RS:0;jenkins-hbase4:34579] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS, ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS] on shutdown 2023-04-12 02:55:52,906 INFO [RS:2;jenkins-hbase4:43091] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS, ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS] on shutdown 2023-04-12 02:55:52,907 INFO [RS:2;jenkins-hbase4:43091] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-04-12 02:55:52,907 INFO [RS:0;jenkins-hbase4:34579] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-04-12 02:55:52,907 INFO [RS:0;jenkins-hbase4:34579] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-04-12 02:55:52,907 INFO [RS:0;jenkins-hbase4:34579] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-04-12 02:55:52,907 INFO [RS:2;jenkins-hbase4:43091] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-04-12 02:55:52,907 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-04-12 02:55:52,907 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-04-12 02:55:52,909 INFO [RS:0;jenkins-hbase4:34579] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:34579 2023-04-12 02:55:52,908 INFO [RS:2;jenkins-hbase4:43091] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-04-12 02:55:52,910 INFO [RS:2;jenkins-hbase4:43091] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:43091 2023-04-12 02:55:52,913 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=78 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/hbase/namespace/1f255e980a31bb266f2206a30aa2ae3f/.tmp/info/fb822ccd51244db8884e71cac0609fc5 2023-04-12 02:55:52,915 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:52,916 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:52,916 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-04-12 02:55:52,916 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,43091,1681268031804 2023-04-12 02:55:52,916 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-04-12 02:55:52,916 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-04-12 02:55:52,916 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-04-12 02:55:52,916 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:52,916 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:52,916 ERROR [Listener at localhost/46667-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@b88814f rejected from java.util.concurrent.ThreadPoolExecutor@fb704e6[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 312] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-12 02:55:52,917 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,34579,1681268031719 2023-04-12 02:55:52,917 ERROR [Listener at localhost/46667-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@74567243 rejected from java.util.concurrent.ThreadPoolExecutor@7069f811[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 312] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-12 02:55:52,917 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=71.29 KB at sequenceid=210 (bloomFilter=false), to=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/hbase/meta/1588230740/.tmp/info/e763d8f700ca47d6bdbe0e10587c2dff 2023-04-12 02:55:52,919 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/hbase/namespace/1f255e980a31bb266f2206a30aa2ae3f/.tmp/info/fb822ccd51244db8884e71cac0609fc5 as hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/hbase/namespace/1f255e980a31bb266f2206a30aa2ae3f/info/fb822ccd51244db8884e71cac0609fc5 2023-04-12 02:55:52,919 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase4.apache.org,34579,1681268031719] 2023-04-12 02:55:52,919 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase4.apache.org,34579,1681268031719; numProcessing=1 2023-04-12 02:55:52,920 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for e763d8f700ca47d6bdbe0e10587c2dff 2023-04-12 02:55:52,920 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,34579,1681268031719 already deleted, retry=false 2023-04-12 02:55:52,920 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase4.apache.org,34579,1681268031719 expired; onlineServers=2 2023-04-12 02:55:52,921 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase4.apache.org,43091,1681268031804] 2023-04-12 02:55:52,921 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase4.apache.org,43091,1681268031804; numProcessing=2 2023-04-12 02:55:52,922 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,43091,1681268031804 already deleted, retry=false 2023-04-12 02:55:52,922 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase4.apache.org,43091,1681268031804 expired; onlineServers=1 2023-04-12 02:55:52,922 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/hbase/namespace/1f255e980a31bb266f2206a30aa2ae3f/info/fb822ccd51244db8884e71cac0609fc5, entries=2, sequenceid=6, filesize=4.8 K 2023-04-12 02:55:52,923 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~78 B/78, heapSize ~472 B/472, currentSize=0 B/0 for 1f255e980a31bb266f2206a30aa2ae3f in 187ms, sequenceid=6, compaction requested=false 2023-04-12 02:55:52,927 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/hbase/namespace/1f255e980a31bb266f2206a30aa2ae3f/recovered.edits/9.seqid, newMaxSeqId=9, maxSeqId=1 2023-04-12 02:55:52,928 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro 2023-04-12 02:55:52,928 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:namespace,,1681268033852.1f255e980a31bb266f2206a30aa2ae3f. 2023-04-12 02:55:52,928 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 1f255e980a31bb266f2206a30aa2ae3f: 2023-04-12 02:55:52,929 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1681268033852.1f255e980a31bb266f2206a30aa2ae3f. 2023-04-12 02:55:52,934 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.58 KB at sequenceid=210 (bloomFilter=false), to=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/hbase/meta/1588230740/.tmp/rep_barrier/49e471f871fb409a86f2083919b61671 2023-04-12 02:55:52,937 DEBUG [RS:1;jenkins-hbase4:44679] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-04-12 02:55:52,937 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 49e471f871fb409a86f2083919b61671 2023-04-12 02:55:53,018 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-12 02:55:53,018 INFO [RS:2;jenkins-hbase4:43091] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,43091,1681268031804; zookeeper connection closed. 2023-04-12 02:55:53,018 ERROR [Listener at localhost/46667-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@34786e6b rejected from java.util.concurrent.ThreadPoolExecutor@fb704e6[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 312] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-12 02:55:53,018 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:43091-0x101764a22930003, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-12 02:55:53,018 ERROR [Listener at localhost/46667-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@4c92b8f6 rejected from java.util.concurrent.ThreadPoolExecutor@fb704e6[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 312] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-12 02:55:53,018 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@5dc70d93] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@5dc70d93 2023-04-12 02:55:53,118 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-12 02:55:53,118 INFO [RS:0;jenkins-hbase4:34579] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,34579,1681268031719; zookeeper connection closed. 2023-04-12 02:55:53,118 ERROR [Listener at localhost/46667-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@4317ef2f rejected from java.util.concurrent.ThreadPoolExecutor@7069f811[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 312] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-12 02:55:53,118 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@5cc35a28] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@5cc35a28 2023-04-12 02:55:53,118 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:34579-0x101764a22930001, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-12 02:55:53,119 ERROR [Listener at localhost/46667-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@3b828087 rejected from java.util.concurrent.ThreadPoolExecutor@7069f811[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 312] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-12 02:55:53,137 DEBUG [RS:1;jenkins-hbase4:44679] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-04-12 02:55:53,232 DEBUG [regionserver/jenkins-hbase4:0.Chore.1] throttle.PressureAwareCompactionThroughputController(103): CompactionPressure is 0.0, tune throughput to 50.00 MB/second 2023-04-12 02:55:53,232 INFO [regionserver/jenkins-hbase4:0.Chore.1] hbase.ScheduledChore(146): Chore: CompactedHFilesCleaner was stopped 2023-04-12 02:55:53,232 INFO [regionserver/jenkins-hbase4:0.Chore.1] hbase.ScheduledChore(146): Chore: CompactionChecker was stopped 2023-04-12 02:55:53,232 INFO [regionserver/jenkins-hbase4:0.Chore.1] hbase.ScheduledChore(146): Chore: MemstoreFlusherChore was stopped 2023-04-12 02:55:53,337 DEBUG [RS:1;jenkins-hbase4:44679] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-04-12 02:55:53,347 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=5.53 KB at sequenceid=210 (bloomFilter=false), to=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/hbase/meta/1588230740/.tmp/table/60636c6235264ea7bddf32601806a146 2023-04-12 02:55:53,350 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 60636c6235264ea7bddf32601806a146 2023-04-12 02:55:53,351 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/hbase/meta/1588230740/.tmp/info/e763d8f700ca47d6bdbe0e10587c2dff as hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/hbase/meta/1588230740/info/e763d8f700ca47d6bdbe0e10587c2dff 2023-04-12 02:55:53,354 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for e763d8f700ca47d6bdbe0e10587c2dff 2023-04-12 02:55:53,354 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/hbase/meta/1588230740/info/e763d8f700ca47d6bdbe0e10587c2dff, entries=59, sequenceid=210, filesize=11.1 K 2023-04-12 02:55:53,355 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/hbase/meta/1588230740/.tmp/rep_barrier/49e471f871fb409a86f2083919b61671 as hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/hbase/meta/1588230740/rep_barrier/49e471f871fb409a86f2083919b61671 2023-04-12 02:55:53,358 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 49e471f871fb409a86f2083919b61671 2023-04-12 02:55:53,358 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/hbase/meta/1588230740/rep_barrier/49e471f871fb409a86f2083919b61671, entries=24, sequenceid=210, filesize=7.5 K 2023-04-12 02:55:53,358 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/hbase/meta/1588230740/.tmp/table/60636c6235264ea7bddf32601806a146 as hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/hbase/meta/1588230740/table/60636c6235264ea7bddf32601806a146 2023-04-12 02:55:53,361 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 60636c6235264ea7bddf32601806a146 2023-04-12 02:55:53,361 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/hbase/meta/1588230740/table/60636c6235264ea7bddf32601806a146, entries=36, sequenceid=210, filesize=8.0 K 2023-04-12 02:55:53,362 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~79.40 KB/81308, heapSize ~125.73 KB/128744, currentSize=0 B/0 for 1588230740 in 626ms, sequenceid=210, compaction requested=false 2023-04-12 02:55:53,370 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/data/hbase/meta/1588230740/recovered.edits/213.seqid, newMaxSeqId=213, maxSeqId=1 2023-04-12 02:55:53,370 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro 2023-04-12 02:55:53,370 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-04-12 02:55:53,371 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-04-12 02:55:53,371 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-04-12 02:55:53,371 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-04-12 02:55:53,537 INFO [RS:1;jenkins-hbase4:44679] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,44679,1681268031772; all regions closed. 2023-04-12 02:55:53,542 DEBUG [RS:1;jenkins-hbase4:44679] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/oldWALs 2023-04-12 02:55:53,542 INFO [RS:1;jenkins-hbase4:44679] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C44679%2C1681268031772.meta:.meta(num 1681268033574) 2023-04-12 02:55:53,547 DEBUG [RS:1;jenkins-hbase4:44679] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/oldWALs 2023-04-12 02:55:53,548 INFO [RS:1;jenkins-hbase4:44679] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C44679%2C1681268031772:(num 1681268033369) 2023-04-12 02:55:53,548 DEBUG [RS:1;jenkins-hbase4:44679] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-12 02:55:53,548 INFO [RS:1;jenkins-hbase4:44679] regionserver.LeaseManager(133): Closed leases 2023-04-12 02:55:53,548 INFO [RS:1;jenkins-hbase4:44679] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS] on shutdown 2023-04-12 02:55:53,548 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-04-12 02:55:53,549 INFO [RS:1;jenkins-hbase4:44679] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:44679 2023-04-12 02:55:53,550 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-04-12 02:55:53,550 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,44679,1681268031772 2023-04-12 02:55:53,550 ERROR [Listener at localhost/46667-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@7876c097 rejected from java.util.concurrent.ThreadPoolExecutor@7c495b41[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 313] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-12 02:55:53,552 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase4.apache.org,44679,1681268031772] 2023-04-12 02:55:53,552 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase4.apache.org,44679,1681268031772; numProcessing=3 2023-04-12 02:55:53,553 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,44679,1681268031772 already deleted, retry=false 2023-04-12 02:55:53,553 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase4.apache.org,44679,1681268031772 expired; onlineServers=0 2023-04-12 02:55:53,553 INFO [RegionServerTracker-0] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,40281,1681268030476' ***** 2023-04-12 02:55:53,553 INFO [RegionServerTracker-0] regionserver.HRegionServer(2310): STOPPED: Cluster shutdown set; onlineServer=0 2023-04-12 02:55:53,554 DEBUG [M:0;jenkins-hbase4:40281] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@35ff3fe1, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-04-12 02:55:53,554 INFO [M:0;jenkins-hbase4:40281] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,40281,1681268030476 2023-04-12 02:55:53,554 INFO [M:0;jenkins-hbase4:40281] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,40281,1681268030476; all regions closed. 2023-04-12 02:55:53,554 DEBUG [M:0;jenkins-hbase4:40281] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-12 02:55:53,554 DEBUG [M:0;jenkins-hbase4:40281] cleaner.LogCleaner(198): Cancelling LogCleaner 2023-04-12 02:55:53,554 WARN [OldWALsCleaner-0] cleaner.LogCleaner(186): Interrupted while cleaning old WALs, will try to clean it next round. Exiting. 2023-04-12 02:55:53,554 DEBUG [M:0;jenkins-hbase4:40281] cleaner.HFileCleaner(317): Stopping file delete threads 2023-04-12 02:55:53,554 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1681268033011] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1681268033011,5,FailOnTimeoutGroup] 2023-04-12 02:55:53,555 INFO [M:0;jenkins-hbase4:40281] master.MasterMobCompactionThread(168): Waiting for Mob Compaction Thread to finish... 2023-04-12 02:55:53,554 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1681268033011] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1681268033011,5,FailOnTimeoutGroup] 2023-04-12 02:55:53,555 INFO [M:0;jenkins-hbase4:40281] master.MasterMobCompactionThread(168): Waiting for Region Server Mob Compaction Thread to finish... 2023-04-12 02:55:53,555 INFO [M:0;jenkins-hbase4:40281] hbase.ChoreService(369): Chore service for: master/jenkins-hbase4:0 had [] on shutdown 2023-04-12 02:55:53,555 DEBUG [M:0;jenkins-hbase4:40281] master.HMaster(1512): Stopping service threads 2023-04-12 02:55:53,555 INFO [M:0;jenkins-hbase4:40281] procedure2.RemoteProcedureDispatcher(118): Stopping procedure remote dispatcher 2023-04-12 02:55:53,555 ERROR [M:0;jenkins-hbase4:40281] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] Thread[HFileArchiver-13,5,PEWorkerGroup] Thread[HFileArchiver-14,5,PEWorkerGroup] Thread[HFileArchiver-15,5,PEWorkerGroup] Thread[HFileArchiver-16,5,PEWorkerGroup] 2023-04-12 02:55:53,555 INFO [M:0;jenkins-hbase4:40281] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-04-12 02:55:53,555 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-04-12 02:55:53,555 DEBUG [normalizer-worker-0] normalizer.RegionNormalizerWorker(174): interrupt detected. terminating. 2023-04-12 02:55:53,555 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-12 02:55:53,556 DEBUG [M:0;jenkins-hbase4:40281] zookeeper.ZKUtil(398): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2023-04-12 02:55:53,556 WARN [M:0;jenkins-hbase4:40281] master.ActiveMasterManager(326): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2023-04-12 02:55:53,556 INFO [M:0;jenkins-hbase4:40281] assignment.AssignmentManager(315): Stopping assignment manager 2023-04-12 02:55:53,556 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-04-12 02:55:53,556 INFO [M:0;jenkins-hbase4:40281] region.MasterRegion(167): Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=false 2023-04-12 02:55:53,556 DEBUG [M:0;jenkins-hbase4:40281] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-04-12 02:55:53,556 INFO [M:0;jenkins-hbase4:40281] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-12 02:55:53,556 DEBUG [M:0;jenkins-hbase4:40281] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-12 02:55:53,556 DEBUG [M:0;jenkins-hbase4:40281] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-04-12 02:55:53,556 DEBUG [M:0;jenkins-hbase4:40281] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-12 02:55:53,556 INFO [M:0;jenkins-hbase4:40281] regionserver.HRegion(2745): Flushing 1595e783b53d99cd5eef43b6debb2682 1/1 column families, dataSize=608.15 KB heapSize=719.04 KB 2023-04-12 02:55:53,564 WARN [IPC Server handler 3 on default port 44691] blockmanagement.BlockPlacementPolicyDefault(446): Failed to place enough replicas, still in need of 1 to reach 3 (unavailableStorages=[], storagePolicy=BlockStoragePolicy{HOT:7, storageTypes=[DISK], creationFallbacks=[], replicationFallbacks=[ARCHIVE]}, newBlock=true) For more information, please enable DEBUG log level on org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicy and org.apache.hadoop.net.NetworkTopology 2023-04-12 02:55:53,564 WARN [IPC Server handler 3 on default port 44691] protocol.BlockStoragePolicy(161): Failed to place enough replicas: expected size is 1 but only 0 storage types can be selected (replication=3, selected=[], unavailable=[DISK], removed=[DISK], policy=BlockStoragePolicy{HOT:7, storageTypes=[DISK], creationFallbacks=[], replicationFallbacks=[ARCHIVE]}) 2023-04-12 02:55:53,564 WARN [IPC Server handler 3 on default port 44691] blockmanagement.BlockPlacementPolicyDefault(446): Failed to place enough replicas, still in need of 1 to reach 3 (unavailableStorages=[DISK], storagePolicy=BlockStoragePolicy{HOT:7, storageTypes=[DISK], creationFallbacks=[], replicationFallbacks=[ARCHIVE]}, newBlock=true) All required storage types are unavailable: unavailableStorages=[DISK], storagePolicy=BlockStoragePolicy{HOT:7, storageTypes=[DISK], creationFallbacks=[], replicationFallbacks=[ARCHIVE]} 2023-04-12 02:55:53,569 INFO [M:0;jenkins-hbase4:40281] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=608.15 KB at sequenceid=1282 (bloomFilter=true), to=hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/cbfceff0636e4b9eb4067dec99770671 2023-04-12 02:55:53,572 INFO [M:0;jenkins-hbase4:40281] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for cbfceff0636e4b9eb4067dec99770671 2023-04-12 02:55:53,573 DEBUG [M:0;jenkins-hbase4:40281] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/cbfceff0636e4b9eb4067dec99770671 as hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/cbfceff0636e4b9eb4067dec99770671 2023-04-12 02:55:53,576 INFO [M:0;jenkins-hbase4:40281] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for cbfceff0636e4b9eb4067dec99770671 2023-04-12 02:55:53,576 INFO [M:0;jenkins-hbase4:40281] regionserver.HStore(1080): Added hdfs://localhost:44691/user/jenkins/test-data/cec631fc-627f-51bf-5997-fcb91cfb896f/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/cbfceff0636e4b9eb4067dec99770671, entries=156, sequenceid=1282, filesize=34.9 K 2023-04-12 02:55:53,576 INFO [M:0;jenkins-hbase4:40281] regionserver.HRegion(2948): Finished flush of dataSize ~608.15 KB/622746, heapSize ~719.02 KB/736280, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 20ms, sequenceid=1282, compaction requested=false 2023-04-12 02:55:53,579 INFO [M:0;jenkins-hbase4:40281] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-12 02:55:53,579 DEBUG [M:0;jenkins-hbase4:40281] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-04-12 02:55:53,583 INFO [M:0;jenkins-hbase4:40281] flush.MasterFlushTableProcedureManager(83): stop: server shutting down. 2023-04-12 02:55:53,583 INFO [master:store-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-04-12 02:55:53,583 INFO [M:0;jenkins-hbase4:40281] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:40281 2023-04-12 02:55:53,584 DEBUG [M:0;jenkins-hbase4:40281] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase4.apache.org,40281,1681268030476 already deleted, retry=false 2023-04-12 02:55:53,652 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-12 02:55:53,653 INFO [RS:1;jenkins-hbase4:44679] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,44679,1681268031772; zookeeper connection closed. 2023-04-12 02:55:53,653 ERROR [Listener at localhost/46667-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@4684550c rejected from java.util.concurrent.ThreadPoolExecutor@7c495b41[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 313] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-12 02:55:53,653 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): regionserver:44679-0x101764a22930002, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-12 02:55:53,653 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@1b93786a] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@1b93786a 2023-04-12 02:55:53,653 ERROR [Listener at localhost/46667-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@3e9d9636 rejected from java.util.concurrent.ThreadPoolExecutor@7c495b41[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 313] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-12 02:55:53,653 INFO [Listener at localhost/46667] util.JVMClusterUtil(335): Shutdown of 1 master(s) and 3 regionserver(s) complete 2023-04-12 02:55:53,753 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-12 02:55:53,753 INFO [M:0;jenkins-hbase4:40281] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,40281,1681268030476; zookeeper connection closed. 2023-04-12 02:55:53,753 ERROR [Listener at localhost/46667-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@1025a49d rejected from java.util.concurrent.ThreadPoolExecutor@15585090[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 750] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-12 02:55:53,753 DEBUG [Listener at localhost/46667-EventThread] zookeeper.ZKWatcher(600): master:40281-0x101764a22930000, quorum=127.0.0.1:57543, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-12 02:55:53,753 ERROR [Listener at localhost/46667-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@646beade rejected from java.util.concurrent.ThreadPoolExecutor@15585090[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 750] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-12 02:55:53,755 WARN [Listener at localhost/46667] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-04-12 02:55:53,758 INFO [Listener at localhost/46667] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-04-12 02:55:53,862 WARN [BP-1247239137-172.31.14.131-1681268026808 heartbeating to localhost/127.0.0.1:44691] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-04-12 02:55:53,862 WARN [BP-1247239137-172.31.14.131-1681268026808 heartbeating to localhost/127.0.0.1:44691] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-1247239137-172.31.14.131-1681268026808 (Datanode Uuid 569a15e2-91b7-43b5-bb2b-d70e35911578) service to localhost/127.0.0.1:44691 2023-04-12 02:55:53,864 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/22e25865-4487-e2aa-7556-0db30ee3c8ff/cluster_2d9c59b9-d025-040c-6051-62d17e49a4b2/dfs/data/data5/current/BP-1247239137-172.31.14.131-1681268026808] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-04-12 02:55:53,864 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/22e25865-4487-e2aa-7556-0db30ee3c8ff/cluster_2d9c59b9-d025-040c-6051-62d17e49a4b2/dfs/data/data6/current/BP-1247239137-172.31.14.131-1681268026808] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-04-12 02:55:53,866 WARN [Listener at localhost/46667] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-04-12 02:55:53,868 INFO [Listener at localhost/46667] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-04-12 02:55:53,943 WARN [ReplicationMonitor] blockmanagement.BlockPlacementPolicyDefault(446): Failed to place enough replicas, still in need of 1 to reach 3 (unavailableStorages=[], storagePolicy=BlockStoragePolicy{HOT:7, storageTypes=[DISK], creationFallbacks=[], replicationFallbacks=[ARCHIVE]}, newBlock=false) For more information, please enable DEBUG log level on org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicy and org.apache.hadoop.net.NetworkTopology 2023-04-12 02:55:53,943 WARN [ReplicationMonitor] blockmanagement.BlockPlacementPolicyDefault(446): Failed to place enough replicas, still in need of 1 to reach 3 (unavailableStorages=[DISK], storagePolicy=BlockStoragePolicy{HOT:7, storageTypes=[DISK], creationFallbacks=[], replicationFallbacks=[ARCHIVE]}, newBlock=false) For more information, please enable DEBUG log level on org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicy and org.apache.hadoop.net.NetworkTopology 2023-04-12 02:55:53,943 WARN [ReplicationMonitor] protocol.BlockStoragePolicy(161): Failed to place enough replicas: expected size is 1 but only 0 storage types can be selected (replication=3, selected=[], unavailable=[DISK, ARCHIVE], removed=[DISK], policy=BlockStoragePolicy{HOT:7, storageTypes=[DISK], creationFallbacks=[], replicationFallbacks=[ARCHIVE]}) 2023-04-12 02:55:53,943 WARN [ReplicationMonitor] blockmanagement.BlockPlacementPolicyDefault(446): Failed to place enough replicas, still in need of 1 to reach 3 (unavailableStorages=[DISK, ARCHIVE], storagePolicy=BlockStoragePolicy{HOT:7, storageTypes=[DISK], creationFallbacks=[], replicationFallbacks=[ARCHIVE]}, newBlock=false) All required storage types are unavailable: unavailableStorages=[DISK, ARCHIVE], storagePolicy=BlockStoragePolicy{HOT:7, storageTypes=[DISK], creationFallbacks=[], replicationFallbacks=[ARCHIVE]} 2023-04-12 02:55:53,972 WARN [BP-1247239137-172.31.14.131-1681268026808 heartbeating to localhost/127.0.0.1:44691] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-04-12 02:55:53,972 WARN [BP-1247239137-172.31.14.131-1681268026808 heartbeating to localhost/127.0.0.1:44691] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-1247239137-172.31.14.131-1681268026808 (Datanode Uuid 79922b64-644d-4757-a58c-9707ee400c82) service to localhost/127.0.0.1:44691 2023-04-12 02:55:53,973 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/22e25865-4487-e2aa-7556-0db30ee3c8ff/cluster_2d9c59b9-d025-040c-6051-62d17e49a4b2/dfs/data/data3/current/BP-1247239137-172.31.14.131-1681268026808] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-04-12 02:55:53,973 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/22e25865-4487-e2aa-7556-0db30ee3c8ff/cluster_2d9c59b9-d025-040c-6051-62d17e49a4b2/dfs/data/data4/current/BP-1247239137-172.31.14.131-1681268026808] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-04-12 02:55:53,975 WARN [Listener at localhost/46667] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-04-12 02:55:53,980 INFO [Listener at localhost/46667] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-04-12 02:55:54,087 WARN [BP-1247239137-172.31.14.131-1681268026808 heartbeating to localhost/127.0.0.1:44691] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-1247239137-172.31.14.131-1681268026808 (Datanode Uuid ad316322-0da1-41fd-a03f-e3dd2b710c84) service to localhost/127.0.0.1:44691 2023-04-12 02:55:54,089 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/22e25865-4487-e2aa-7556-0db30ee3c8ff/cluster_2d9c59b9-d025-040c-6051-62d17e49a4b2/dfs/data/data1/current/BP-1247239137-172.31.14.131-1681268026808] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-04-12 02:55:54,090 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/22e25865-4487-e2aa-7556-0db30ee3c8ff/cluster_2d9c59b9-d025-040c-6051-62d17e49a4b2/dfs/data/data2/current/BP-1247239137-172.31.14.131-1681268026808] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-04-12 02:55:54,111 INFO [Listener at localhost/46667] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-04-12 02:55:54,299 INFO [Listener at localhost/46667] zookeeper.MiniZooKeeperCluster(344): Shutdown MiniZK cluster with all ZK servers 2023-04-12 02:55:54,526 INFO [Listener at localhost/46667] hbase.HBaseTestingUtility(1293): Minicluster is down