2023-02-11 01:00:46,647 DEBUG [main] hbase.HBaseTestingUtility(342): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e5dd7502-721b-939f-9fad-68d9c52e98bf 2023-02-11 01:00:46,661 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.client.TestAsyncClusterAdminApi2 timeout: 13 mins 2023-02-11 01:00:46,697 INFO [Time-limited test] hbase.ResourceChecker(147): before: client.TestAsyncClusterAdminApi2#testStop Thread=8, OpenFileDescriptor=260, MaxFileDescriptor=60000, SystemLoadAverage=184, ProcessCount=168, AvailableMemoryMB=3766 2023-02-11 01:00:46,703 INFO [Time-limited test] hbase.HBaseTestingUtility(1068): Starting up minicluster with option: StartMiniClusterOption{numMasters=1, masterClass=null, numRegionServers=3, rsPorts=, rsClass=null, numDataNodes=3, dataNodeHosts=null, numZkServers=1, createRootDir=false, createWALDir=false} 2023-02-11 01:00:46,704 INFO [Time-limited test] hbase.HBaseZKTestingUtility(82): Created new mini-cluster data directory: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e5dd7502-721b-939f-9fad-68d9c52e98bf/cluster_57098650-f7a3-dd69-97b5-a6c2a3ffc93f, deleteOnExit=true 2023-02-11 01:00:46,704 INFO [Time-limited test] hbase.HBaseTestingUtility(1082): STARTING DFS 2023-02-11 01:00:46,704 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting test.cache.data to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e5dd7502-721b-939f-9fad-68d9c52e98bf/test.cache.data in system properties and HBase conf 2023-02-11 01:00:46,705 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.tmp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e5dd7502-721b-939f-9fad-68d9c52e98bf/hadoop.tmp.dir in system properties and HBase conf 2023-02-11 01:00:46,705 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.log.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e5dd7502-721b-939f-9fad-68d9c52e98bf/hadoop.log.dir in system properties and HBase conf 2023-02-11 01:00:46,706 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.local.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e5dd7502-721b-939f-9fad-68d9c52e98bf/mapreduce.cluster.local.dir in system properties and HBase conf 2023-02-11 01:00:46,706 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.temp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e5dd7502-721b-939f-9fad-68d9c52e98bf/mapreduce.cluster.temp.dir in system properties and HBase conf 2023-02-11 01:00:46,706 INFO [Time-limited test] hbase.HBaseTestingUtility(759): read short circuit is OFF 2023-02-11 01:00:46,823 WARN [Time-limited test] util.NativeCodeLoader(62): Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2023-02-11 01:00:47,195 DEBUG [Time-limited test] fs.HFileSystem(308): The file system is not a DistributedFileSystem. Skipping on block location reordering 2023-02-11 01:00:47,199 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-labels.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e5dd7502-721b-939f-9fad-68d9c52e98bf/yarn.node-labels.fs-store.root-dir in system properties and HBase conf 2023-02-11 01:00:47,199 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-attribute.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e5dd7502-721b-939f-9fad-68d9c52e98bf/yarn.node-attribute.fs-store.root-dir in system properties and HBase conf 2023-02-11 01:00:47,199 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.log-dirs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e5dd7502-721b-939f-9fad-68d9c52e98bf/yarn.nodemanager.log-dirs in system properties and HBase conf 2023-02-11 01:00:47,200 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e5dd7502-721b-939f-9fad-68d9c52e98bf/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-02-11 01:00:47,200 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.active-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e5dd7502-721b-939f-9fad-68d9c52e98bf/yarn.timeline-service.entity-group-fs-store.active-dir in system properties and HBase conf 2023-02-11 01:00:47,200 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.done-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e5dd7502-721b-939f-9fad-68d9c52e98bf/yarn.timeline-service.entity-group-fs-store.done-dir in system properties and HBase conf 2023-02-11 01:00:47,201 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e5dd7502-721b-939f-9fad-68d9c52e98bf/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-02-11 01:00:47,201 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e5dd7502-721b-939f-9fad-68d9c52e98bf/dfs.journalnode.edits.dir in system properties and HBase conf 2023-02-11 01:00:47,201 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.datanode.shared.file.descriptor.paths to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e5dd7502-721b-939f-9fad-68d9c52e98bf/dfs.datanode.shared.file.descriptor.paths in system properties and HBase conf 2023-02-11 01:00:47,202 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting nfs.dump.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e5dd7502-721b-939f-9fad-68d9c52e98bf/nfs.dump.dir in system properties and HBase conf 2023-02-11 01:00:47,202 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting java.io.tmpdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e5dd7502-721b-939f-9fad-68d9c52e98bf/java.io.tmpdir in system properties and HBase conf 2023-02-11 01:00:47,202 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e5dd7502-721b-939f-9fad-68d9c52e98bf/dfs.journalnode.edits.dir in system properties and HBase conf 2023-02-11 01:00:47,203 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.provided.aliasmap.inmemory.leveldb.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e5dd7502-721b-939f-9fad-68d9c52e98bf/dfs.provided.aliasmap.inmemory.leveldb.dir in system properties and HBase conf 2023-02-11 01:00:47,203 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting fs.s3a.committer.staging.tmp.path to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e5dd7502-721b-939f-9fad-68d9c52e98bf/fs.s3a.committer.staging.tmp.path in system properties and HBase conf Formatting using clusterid: testClusterID 2023-02-11 01:00:47,702 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-02-11 01:00:47,707 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-02-11 01:00:47,971 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties 2023-02-11 01:00:48,115 INFO [Time-limited test] log.Slf4jLog(67): Logging to org.slf4j.impl.Reload4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2023-02-11 01:00:48,129 WARN [Time-limited test] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-02-11 01:00:48,166 INFO [Time-limited test] log.Slf4jLog(67): jetty-6.1.26 2023-02-11 01:00:48,198 INFO [Time-limited test] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/hdfs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e5dd7502-721b-939f-9fad-68d9c52e98bf/java.io.tmpdir/Jetty_localhost_42431_hdfs____wdce7y/webapp 2023-02-11 01:00:48,360 INFO [Time-limited test] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:42431 2023-02-11 01:00:48,371 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-02-11 01:00:48,371 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-02-11 01:00:48,771 WARN [Listener at localhost/44295] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-02-11 01:00:48,834 WARN [Listener at localhost/44295] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-02-11 01:00:48,851 WARN [Listener at localhost/44295] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-02-11 01:00:48,857 INFO [Listener at localhost/44295] log.Slf4jLog(67): jetty-6.1.26 2023-02-11 01:00:48,861 INFO [Listener at localhost/44295] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e5dd7502-721b-939f-9fad-68d9c52e98bf/java.io.tmpdir/Jetty_localhost_43967_datanode____.dztvi6/webapp 2023-02-11 01:00:48,962 INFO [Listener at localhost/44295] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:43967 2023-02-11 01:00:49,298 WARN [Listener at localhost/34661] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-02-11 01:00:49,310 WARN [Listener at localhost/34661] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-02-11 01:00:49,313 WARN [Listener at localhost/34661] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-02-11 01:00:49,316 INFO [Listener at localhost/34661] log.Slf4jLog(67): jetty-6.1.26 2023-02-11 01:00:49,326 INFO [Listener at localhost/34661] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e5dd7502-721b-939f-9fad-68d9c52e98bf/java.io.tmpdir/Jetty_localhost_37365_datanode____31yz3p/webapp 2023-02-11 01:00:49,433 INFO [Listener at localhost/34661] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:37365 2023-02-11 01:00:49,443 WARN [Listener at localhost/40821] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-02-11 01:00:49,453 WARN [Listener at localhost/40821] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-02-11 01:00:49,456 WARN [Listener at localhost/40821] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-02-11 01:00:49,457 INFO [Listener at localhost/40821] log.Slf4jLog(67): jetty-6.1.26 2023-02-11 01:00:49,462 INFO [Listener at localhost/40821] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e5dd7502-721b-939f-9fad-68d9c52e98bf/java.io.tmpdir/Jetty_localhost_37961_datanode____tztuif/webapp 2023-02-11 01:00:49,569 INFO [Listener at localhost/40821] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:37961 2023-02-11 01:00:49,578 WARN [Listener at localhost/46601] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-02-11 01:00:49,770 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xc9f6fd44600a5d26: Processing first storage report for DS-19365098-07a0-4163-8732-27153e4b255b from datanode 0ed6119c-6e7c-46d2-bdd7-f9fb0be3812e 2023-02-11 01:00:49,771 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xc9f6fd44600a5d26: from storage DS-19365098-07a0-4163-8732-27153e4b255b node DatanodeRegistration(127.0.0.1:37515, datanodeUuid=0ed6119c-6e7c-46d2-bdd7-f9fb0be3812e, infoPort=40939, infoSecurePort=0, ipcPort=34661, storageInfo=lv=-57;cid=testClusterID;nsid=483479740;c=1676077247778), blocks: 0, hasStaleStorage: true, processing time: 2 msecs, invalidatedBlocks: 0 2023-02-11 01:00:49,771 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xc0af4208433d9aeb: Processing first storage report for DS-8f401005-340e-4621-a18e-f461eb4d4ab2 from datanode 5d966976-4df8-4674-b9ce-a32041365514 2023-02-11 01:00:49,771 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xc0af4208433d9aeb: from storage DS-8f401005-340e-4621-a18e-f461eb4d4ab2 node DatanodeRegistration(127.0.0.1:39157, datanodeUuid=5d966976-4df8-4674-b9ce-a32041365514, infoPort=38825, infoSecurePort=0, ipcPort=40821, storageInfo=lv=-57;cid=testClusterID;nsid=483479740;c=1676077247778), blocks: 0, hasStaleStorage: true, processing time: 0 msecs, invalidatedBlocks: 0 2023-02-11 01:00:49,772 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xb5a6ea3672e899f0: Processing first storage report for DS-9ebc61aa-9b9c-41d9-abc9-b4b9b6de6337 from datanode cdf5a302-30b0-4180-97ad-ca2e2d791bac 2023-02-11 01:00:49,772 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xb5a6ea3672e899f0: from storage DS-9ebc61aa-9b9c-41d9-abc9-b4b9b6de6337 node DatanodeRegistration(127.0.0.1:39915, datanodeUuid=cdf5a302-30b0-4180-97ad-ca2e2d791bac, infoPort=36129, infoSecurePort=0, ipcPort=46601, storageInfo=lv=-57;cid=testClusterID;nsid=483479740;c=1676077247778), blocks: 0, hasStaleStorage: true, processing time: 0 msecs, invalidatedBlocks: 0 2023-02-11 01:00:49,772 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xc9f6fd44600a5d26: Processing first storage report for DS-401e60de-e1a0-42ed-8da7-45683ea42aa0 from datanode 0ed6119c-6e7c-46d2-bdd7-f9fb0be3812e 2023-02-11 01:00:49,772 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xc9f6fd44600a5d26: from storage DS-401e60de-e1a0-42ed-8da7-45683ea42aa0 node DatanodeRegistration(127.0.0.1:37515, datanodeUuid=0ed6119c-6e7c-46d2-bdd7-f9fb0be3812e, infoPort=40939, infoSecurePort=0, ipcPort=34661, storageInfo=lv=-57;cid=testClusterID;nsid=483479740;c=1676077247778), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-02-11 01:00:49,772 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xc0af4208433d9aeb: Processing first storage report for DS-98e6b155-0579-443f-a792-7c624f9da933 from datanode 5d966976-4df8-4674-b9ce-a32041365514 2023-02-11 01:00:49,772 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xc0af4208433d9aeb: from storage DS-98e6b155-0579-443f-a792-7c624f9da933 node DatanodeRegistration(127.0.0.1:39157, datanodeUuid=5d966976-4df8-4674-b9ce-a32041365514, infoPort=38825, infoSecurePort=0, ipcPort=40821, storageInfo=lv=-57;cid=testClusterID;nsid=483479740;c=1676077247778), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-02-11 01:00:49,773 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xb5a6ea3672e899f0: Processing first storage report for DS-2f49144f-f495-4bb0-a462-1441c207184a from datanode cdf5a302-30b0-4180-97ad-ca2e2d791bac 2023-02-11 01:00:49,773 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xb5a6ea3672e899f0: from storage DS-2f49144f-f495-4bb0-a462-1441c207184a node DatanodeRegistration(127.0.0.1:39915, datanodeUuid=cdf5a302-30b0-4180-97ad-ca2e2d791bac, infoPort=36129, infoSecurePort=0, ipcPort=46601, storageInfo=lv=-57;cid=testClusterID;nsid=483479740;c=1676077247778), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-02-11 01:00:49,962 DEBUG [Listener at localhost/46601] hbase.HBaseTestingUtility(649): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e5dd7502-721b-939f-9fad-68d9c52e98bf 2023-02-11 01:00:50,020 INFO [Listener at localhost/46601] zookeeper.MiniZooKeeperCluster(258): Started connectionTimeout=30000, dir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e5dd7502-721b-939f-9fad-68d9c52e98bf/cluster_57098650-f7a3-dd69-97b5-a6c2a3ffc93f/zookeeper_0, clientPort=63347, secureClientPort=-1, dataDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e5dd7502-721b-939f-9fad-68d9c52e98bf/cluster_57098650-f7a3-dd69-97b5-a6c2a3ffc93f/zookeeper_0/version-2, dataDirSize=424 dataLogDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e5dd7502-721b-939f-9fad-68d9c52e98bf/cluster_57098650-f7a3-dd69-97b5-a6c2a3ffc93f/zookeeper_0/version-2, dataLogSize=424 tickTime=2000, maxClientCnxns=300, minSessionTimeout=4000, maxSessionTimeout=40000, serverId=0 2023-02-11 01:00:50,034 INFO [Listener at localhost/46601] zookeeper.MiniZooKeeperCluster(283): Started MiniZooKeeperCluster and ran 'stat' on client port=63347 2023-02-11 01:00:50,041 INFO [Listener at localhost/46601] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-11 01:00:50,043 INFO [Listener at localhost/46601] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-11 01:00:50,706 INFO [Listener at localhost/46601] util.FSUtils(479): Created version file at hdfs://localhost:44295/user/jenkins/test-data/593564ae-939c-6914-503a-94843f3f7058 with version=8 2023-02-11 01:00:50,707 INFO [Listener at localhost/46601] hbase.HBaseTestingUtility(1406): Setting hbase.fs.tmp.dir to hdfs://localhost:44295/user/jenkins/test-data/593564ae-939c-6914-503a-94843f3f7058/hbase-staging 2023-02-11 01:00:51,000 INFO [Listener at localhost/46601] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-02-11 01:00:51,454 INFO [Listener at localhost/46601] client.ConnectionUtils(127): master/jenkins-hbase4:0 server-side Connection retries=6 2023-02-11 01:00:51,485 INFO [Listener at localhost/46601] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-11 01:00:51,486 INFO [Listener at localhost/46601] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-02-11 01:00:51,486 INFO [Listener at localhost/46601] ipc.RWQueueRpcExecutor(105): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-02-11 01:00:51,486 INFO [Listener at localhost/46601] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-11 01:00:51,486 INFO [Listener at localhost/46601] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-02-11 01:00:51,625 INFO [Listener at localhost/46601] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-02-11 01:00:51,704 DEBUG [Listener at localhost/46601] util.ClassSize(228): Using Unsafe to estimate memory layout 2023-02-11 01:00:51,796 INFO [Listener at localhost/46601] ipc.NettyRpcServer(120): Bind to /172.31.14.131:33145 2023-02-11 01:00:51,807 INFO [Listener at localhost/46601] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-11 01:00:51,809 INFO [Listener at localhost/46601] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-11 01:00:51,830 INFO [Listener at localhost/46601] zookeeper.RecoverableZooKeeper(93): Process identifier=master:33145 connecting to ZooKeeper ensemble=127.0.0.1:63347 2023-02-11 01:00:51,877 DEBUG [Listener at localhost/46601-EventThread] zookeeper.ZKWatcher(600): master:331450x0, quorum=127.0.0.1:63347, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-02-11 01:00:51,879 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): master:33145-0x10040e51f9c0000 connected 2023-02-11 01:00:51,908 DEBUG [Listener at localhost/46601] zookeeper.ZKUtil(164): master:33145-0x10040e51f9c0000, quorum=127.0.0.1:63347, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-11 01:00:51,909 DEBUG [Listener at localhost/46601] zookeeper.ZKUtil(164): master:33145-0x10040e51f9c0000, quorum=127.0.0.1:63347, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-11 01:00:51,913 DEBUG [Listener at localhost/46601] zookeeper.ZKUtil(164): master:33145-0x10040e51f9c0000, quorum=127.0.0.1:63347, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-02-11 01:00:51,920 DEBUG [Listener at localhost/46601] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=33145 2023-02-11 01:00:51,921 DEBUG [Listener at localhost/46601] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=33145 2023-02-11 01:00:51,921 DEBUG [Listener at localhost/46601] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=33145 2023-02-11 01:00:51,922 DEBUG [Listener at localhost/46601] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=33145 2023-02-11 01:00:51,922 DEBUG [Listener at localhost/46601] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=33145 2023-02-11 01:00:51,928 INFO [Listener at localhost/46601] master.HMaster(439): hbase.rootdir=hdfs://localhost:44295/user/jenkins/test-data/593564ae-939c-6914-503a-94843f3f7058, hbase.cluster.distributed=false 2023-02-11 01:00:51,993 INFO [Listener at localhost/46601] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=6 2023-02-11 01:00:51,994 INFO [Listener at localhost/46601] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-11 01:00:51,994 INFO [Listener at localhost/46601] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-02-11 01:00:51,994 INFO [Listener at localhost/46601] ipc.RWQueueRpcExecutor(105): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-02-11 01:00:51,994 INFO [Listener at localhost/46601] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-11 01:00:51,994 INFO [Listener at localhost/46601] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-02-11 01:00:51,999 INFO [Listener at localhost/46601] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-02-11 01:00:52,002 INFO [Listener at localhost/46601] ipc.NettyRpcServer(120): Bind to /172.31.14.131:41571 2023-02-11 01:00:52,004 INFO [Listener at localhost/46601] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-02-11 01:00:52,011 DEBUG [Listener at localhost/46601] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-02-11 01:00:52,012 INFO [Listener at localhost/46601] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-11 01:00:52,014 INFO [Listener at localhost/46601] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-11 01:00:52,015 INFO [Listener at localhost/46601] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:41571 connecting to ZooKeeper ensemble=127.0.0.1:63347 2023-02-11 01:00:52,019 DEBUG [Listener at localhost/46601-EventThread] zookeeper.ZKWatcher(600): regionserver:415710x0, quorum=127.0.0.1:63347, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-02-11 01:00:52,020 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:41571-0x10040e51f9c0001 connected 2023-02-11 01:00:52,020 DEBUG [Listener at localhost/46601] zookeeper.ZKUtil(164): regionserver:41571-0x10040e51f9c0001, quorum=127.0.0.1:63347, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-11 01:00:52,022 DEBUG [Listener at localhost/46601] zookeeper.ZKUtil(164): regionserver:41571-0x10040e51f9c0001, quorum=127.0.0.1:63347, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-11 01:00:52,023 DEBUG [Listener at localhost/46601] zookeeper.ZKUtil(164): regionserver:41571-0x10040e51f9c0001, quorum=127.0.0.1:63347, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-02-11 01:00:52,024 DEBUG [Listener at localhost/46601] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=41571 2023-02-11 01:00:52,024 DEBUG [Listener at localhost/46601] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=41571 2023-02-11 01:00:52,024 DEBUG [Listener at localhost/46601] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=41571 2023-02-11 01:00:52,025 DEBUG [Listener at localhost/46601] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=41571 2023-02-11 01:00:52,025 DEBUG [Listener at localhost/46601] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=41571 2023-02-11 01:00:52,038 INFO [Listener at localhost/46601] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=6 2023-02-11 01:00:52,038 INFO [Listener at localhost/46601] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-11 01:00:52,038 INFO [Listener at localhost/46601] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-02-11 01:00:52,039 INFO [Listener at localhost/46601] ipc.RWQueueRpcExecutor(105): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-02-11 01:00:52,039 INFO [Listener at localhost/46601] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-11 01:00:52,039 INFO [Listener at localhost/46601] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-02-11 01:00:52,039 INFO [Listener at localhost/46601] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-02-11 01:00:52,041 INFO [Listener at localhost/46601] ipc.NettyRpcServer(120): Bind to /172.31.14.131:44999 2023-02-11 01:00:52,042 INFO [Listener at localhost/46601] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-02-11 01:00:52,043 DEBUG [Listener at localhost/46601] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-02-11 01:00:52,044 INFO [Listener at localhost/46601] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-11 01:00:52,046 INFO [Listener at localhost/46601] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-11 01:00:52,048 INFO [Listener at localhost/46601] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:44999 connecting to ZooKeeper ensemble=127.0.0.1:63347 2023-02-11 01:00:52,052 DEBUG [Listener at localhost/46601-EventThread] zookeeper.ZKWatcher(600): regionserver:449990x0, quorum=127.0.0.1:63347, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-02-11 01:00:52,053 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:44999-0x10040e51f9c0002 connected 2023-02-11 01:00:52,053 DEBUG [Listener at localhost/46601] zookeeper.ZKUtil(164): regionserver:44999-0x10040e51f9c0002, quorum=127.0.0.1:63347, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-11 01:00:52,054 DEBUG [Listener at localhost/46601] zookeeper.ZKUtil(164): regionserver:44999-0x10040e51f9c0002, quorum=127.0.0.1:63347, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-11 01:00:52,055 DEBUG [Listener at localhost/46601] zookeeper.ZKUtil(164): regionserver:44999-0x10040e51f9c0002, quorum=127.0.0.1:63347, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-02-11 01:00:52,055 DEBUG [Listener at localhost/46601] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=44999 2023-02-11 01:00:52,056 DEBUG [Listener at localhost/46601] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=44999 2023-02-11 01:00:52,056 DEBUG [Listener at localhost/46601] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=44999 2023-02-11 01:00:52,056 DEBUG [Listener at localhost/46601] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=44999 2023-02-11 01:00:52,057 DEBUG [Listener at localhost/46601] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=44999 2023-02-11 01:00:52,069 INFO [Listener at localhost/46601] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=6 2023-02-11 01:00:52,070 INFO [Listener at localhost/46601] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-11 01:00:52,070 INFO [Listener at localhost/46601] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-02-11 01:00:52,070 INFO [Listener at localhost/46601] ipc.RWQueueRpcExecutor(105): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-02-11 01:00:52,070 INFO [Listener at localhost/46601] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-11 01:00:52,070 INFO [Listener at localhost/46601] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-02-11 01:00:52,070 INFO [Listener at localhost/46601] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-02-11 01:00:52,071 INFO [Listener at localhost/46601] ipc.NettyRpcServer(120): Bind to /172.31.14.131:42073 2023-02-11 01:00:52,072 INFO [Listener at localhost/46601] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-02-11 01:00:52,073 DEBUG [Listener at localhost/46601] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-02-11 01:00:52,074 INFO [Listener at localhost/46601] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-11 01:00:52,075 INFO [Listener at localhost/46601] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-11 01:00:52,076 INFO [Listener at localhost/46601] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:42073 connecting to ZooKeeper ensemble=127.0.0.1:63347 2023-02-11 01:00:52,080 DEBUG [Listener at localhost/46601-EventThread] zookeeper.ZKWatcher(600): regionserver:420730x0, quorum=127.0.0.1:63347, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-02-11 01:00:52,081 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:42073-0x10040e51f9c0003 connected 2023-02-11 01:00:52,081 DEBUG [Listener at localhost/46601] zookeeper.ZKUtil(164): regionserver:42073-0x10040e51f9c0003, quorum=127.0.0.1:63347, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-11 01:00:52,082 DEBUG [Listener at localhost/46601] zookeeper.ZKUtil(164): regionserver:42073-0x10040e51f9c0003, quorum=127.0.0.1:63347, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-11 01:00:52,082 DEBUG [Listener at localhost/46601] zookeeper.ZKUtil(164): regionserver:42073-0x10040e51f9c0003, quorum=127.0.0.1:63347, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-02-11 01:00:52,083 DEBUG [Listener at localhost/46601] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=42073 2023-02-11 01:00:52,083 DEBUG [Listener at localhost/46601] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=42073 2023-02-11 01:00:52,083 DEBUG [Listener at localhost/46601] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=42073 2023-02-11 01:00:52,084 DEBUG [Listener at localhost/46601] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=42073 2023-02-11 01:00:52,084 DEBUG [Listener at localhost/46601] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=42073 2023-02-11 01:00:52,086 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2158): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase4.apache.org,33145,1676077250844 2023-02-11 01:00:52,096 DEBUG [Listener at localhost/46601-EventThread] zookeeper.ZKWatcher(600): master:33145-0x10040e51f9c0000, quorum=127.0.0.1:63347, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-02-11 01:00:52,097 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:33145-0x10040e51f9c0000, quorum=127.0.0.1:63347, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase4.apache.org,33145,1676077250844 2023-02-11 01:00:52,114 DEBUG [Listener at localhost/46601-EventThread] zookeeper.ZKWatcher(600): regionserver:41571-0x10040e51f9c0001, quorum=127.0.0.1:63347, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-02-11 01:00:52,114 DEBUG [Listener at localhost/46601-EventThread] zookeeper.ZKWatcher(600): regionserver:44999-0x10040e51f9c0002, quorum=127.0.0.1:63347, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-02-11 01:00:52,114 DEBUG [Listener at localhost/46601-EventThread] zookeeper.ZKWatcher(600): master:33145-0x10040e51f9c0000, quorum=127.0.0.1:63347, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-02-11 01:00:52,114 DEBUG [Listener at localhost/46601-EventThread] zookeeper.ZKWatcher(600): regionserver:42073-0x10040e51f9c0003, quorum=127.0.0.1:63347, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-02-11 01:00:52,115 DEBUG [Listener at localhost/46601-EventThread] zookeeper.ZKWatcher(600): master:33145-0x10040e51f9c0000, quorum=127.0.0.1:63347, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-11 01:00:52,116 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:33145-0x10040e51f9c0000, quorum=127.0.0.1:63347, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-02-11 01:00:52,118 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(224): Deleting ZNode for /hbase/backup-masters/jenkins-hbase4.apache.org,33145,1676077250844 from backup master directory 2023-02-11 01:00:52,118 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:33145-0x10040e51f9c0000, quorum=127.0.0.1:63347, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-02-11 01:00:52,122 DEBUG [Listener at localhost/46601-EventThread] zookeeper.ZKWatcher(600): master:33145-0x10040e51f9c0000, quorum=127.0.0.1:63347, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase4.apache.org,33145,1676077250844 2023-02-11 01:00:52,122 DEBUG [Listener at localhost/46601-EventThread] zookeeper.ZKWatcher(600): master:33145-0x10040e51f9c0000, quorum=127.0.0.1:63347, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-02-11 01:00:52,123 WARN [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-02-11 01:00:52,123 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(234): Registered as active master=jenkins-hbase4.apache.org,33145,1676077250844 2023-02-11 01:00:52,126 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating data MemStoreChunkPool with chunk size 2 MB, max count 352, initial count 0 2023-02-11 01:00:52,127 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating index MemStoreChunkPool with chunk size 204.80 KB, max count 391, initial count 0 2023-02-11 01:00:52,218 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] util.FSUtils(628): Created cluster ID file at hdfs://localhost:44295/user/jenkins/test-data/593564ae-939c-6914-503a-94843f3f7058/hbase.id with ID: eb576bd2-5fcf-4742-a837-61e802658a82 2023-02-11 01:00:52,260 INFO [master/jenkins-hbase4:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-11 01:00:52,276 DEBUG [Listener at localhost/46601-EventThread] zookeeper.ZKWatcher(600): master:33145-0x10040e51f9c0000, quorum=127.0.0.1:63347, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-11 01:00:52,324 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x2afa76c5 to 127.0.0.1:63347 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-11 01:00:52,362 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1220a1fa, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-11 01:00:52,384 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-02-11 01:00:52,386 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-02-11 01:00:52,402 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(264): ClientProtocol::create wrong number of arguments, should be hadoop 3.2 or below 2023-02-11 01:00:52,402 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(270): ClientProtocol::create wrong number of arguments, should be hadoop 2.x 2023-02-11 01:00:52,404 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(279): can not find SHOULD_REPLICATE flag, should be hadoop 2.x java.lang.IllegalArgumentException: No enum constant org.apache.hadoop.fs.CreateFlag.SHOULD_REPLICATE at java.lang.Enum.valueOf(Enum.java:238) at org.apache.hadoop.fs.CreateFlag.valueOf(CreateFlag.java:63) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.loadShouldReplicateFlag(FanOutOneBlockAsyncDFSOutputHelper.java:277) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.(FanOutOneBlockAsyncDFSOutputHelper.java:304) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:139) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:849) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2178) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:523) at java.lang.Thread.run(Thread.java:750) 2023-02-11 01:00:52,407 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(243): No decryptEncryptedDataEncryptionKey method in DFSClient, should be hadoop version with HDFS-12396 java.lang.NoSuchMethodException: org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(org.apache.hadoop.fs.FileEncryptionInfo) at java.lang.Class.getDeclaredMethod(Class.java:2130) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelperWithoutHDFS12396(FanOutOneBlockAsyncDFSOutputSaslHelper.java:182) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelper(FanOutOneBlockAsyncDFSOutputSaslHelper.java:241) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.(FanOutOneBlockAsyncDFSOutputSaslHelper.java:252) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:140) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:849) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2178) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:523) at java.lang.Thread.run(Thread.java:750) 2023-02-11 01:00:52,409 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-11 01:00:52,440 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7689): Creating {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, under table dir hdfs://localhost:44295/user/jenkins/test-data/593564ae-939c-6914-503a-94843f3f7058/MasterData/data/master/store-tmp 2023-02-11 01:00:52,491 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(865): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 01:00:52,491 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1603): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-02-11 01:00:52,491 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1625): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-11 01:00:52,492 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1646): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-11 01:00:52,492 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1713): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-02-11 01:00:52,492 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1723): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-11 01:00:52,492 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1837): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-11 01:00:52,492 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1557): Region close journal for 1595e783b53d99cd5eef43b6debb2682: Waiting for close lock at 1676077252491Disabling compacts and flushes for region at 1676077252491Disabling writes for close at 1676077252492 (+1 ms)Writing region close event to WAL at 1676077252492Closed at 1676077252492 2023-02-11 01:00:52,493 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(191): WALDir=hdfs://localhost:44295/user/jenkins/test-data/593564ae-939c-6914-503a-94843f3f7058/MasterData/WALs/jenkins-hbase4.apache.org,33145,1676077250844 2023-02-11 01:00:52,512 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C33145%2C1676077250844, suffix=, logDir=hdfs://localhost:44295/user/jenkins/test-data/593564ae-939c-6914-503a-94843f3f7058/MasterData/WALs/jenkins-hbase4.apache.org,33145,1676077250844, archiveDir=hdfs://localhost:44295/user/jenkins/test-data/593564ae-939c-6914-503a-94843f3f7058/MasterData/oldWALs, maxLogs=10 2023-02-11 01:00:52,566 DEBUG [RS-EventLoopGroup-5-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:37515,DS-19365098-07a0-4163-8732-27153e4b255b,DISK] 2023-02-11 01:00:52,566 DEBUG [RS-EventLoopGroup-5-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:39157,DS-8f401005-340e-4621-a18e-f461eb4d4ab2,DISK] 2023-02-11 01:00:52,566 DEBUG [RS-EventLoopGroup-5-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:39915,DS-9ebc61aa-9b9c-41d9-abc9-b4b9b6de6337,DISK] 2023-02-11 01:00:52,574 DEBUG [RS-EventLoopGroup-5-2] asyncfs.ProtobufDecoder(123): Hadoop 3.2 and below use unshaded protobuf. java.lang.ClassNotFoundException: org.apache.hadoop.thirdparty.protobuf.MessageLite at java.net.URLClassLoader.findClass(URLClassLoader.java:387) at java.lang.ClassLoader.loadClass(ClassLoader.java:418) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) at java.lang.ClassLoader.loadClass(ClassLoader.java:351) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.io.asyncfs.ProtobufDecoder.(ProtobufDecoder.java:118) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.processWriteBlockResponse(FanOutOneBlockAsyncDFSOutputHelper.java:340) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$100(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$4.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:424) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:557) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:185) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.initialize(FanOutOneBlockAsyncDFSOutputHelper.java:418) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$300(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:476) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:471) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:625) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:105) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.fulfillConnectPromise(AbstractEpollChannel.java:653) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:691) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-02-11 01:00:52,637 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/593564ae-939c-6914-503a-94843f3f7058/MasterData/WALs/jenkins-hbase4.apache.org,33145,1676077250844/jenkins-hbase4.apache.org%2C33145%2C1676077250844.1676077252522 2023-02-11 01:00:52,638 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:39915,DS-9ebc61aa-9b9c-41d9-abc9-b4b9b6de6337,DISK], DatanodeInfoWithStorage[127.0.0.1:37515,DS-19365098-07a0-4163-8732-27153e4b255b,DISK], DatanodeInfoWithStorage[127.0.0.1:39157,DS-8f401005-340e-4621-a18e-f461eb4d4ab2,DISK]] 2023-02-11 01:00:52,638 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7850): Opening region: {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''} 2023-02-11 01:00:52,639 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(865): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 01:00:52,642 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7890): checking encryption for 1595e783b53d99cd5eef43b6debb2682 2023-02-11 01:00:52,643 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7893): checking classloading for 1595e783b53d99cd5eef43b6debb2682 2023-02-11 01:00:52,700 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family proc of region 1595e783b53d99cd5eef43b6debb2682 2023-02-11 01:00:52,706 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44295/user/jenkins/test-data/593564ae-939c-6914-503a-94843f3f7058/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc 2023-02-11 01:00:52,738 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc 2023-02-11 01:00:52,752 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(310): Store=1595e783b53d99cd5eef43b6debb2682/proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 01:00:52,757 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:44295/user/jenkins/test-data/593564ae-939c-6914-503a-94843f3f7058/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-02-11 01:00:52,759 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:44295/user/jenkins/test-data/593564ae-939c-6914-503a-94843f3f7058/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-02-11 01:00:52,774 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1054): writing seq id for 1595e783b53d99cd5eef43b6debb2682 2023-02-11 01:00:52,778 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44295/user/jenkins/test-data/593564ae-939c-6914-503a-94843f3f7058/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-11 01:00:52,779 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1071): Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=72800920, jitterRate=0.08481824398040771}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 01:00:52,779 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(964): Region open journal for 1595e783b53d99cd5eef43b6debb2682: Writing region info on filesystem at 1676077252669Initializing all the Stores at 1676077252671 (+2 ms)Instantiating store for column family {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676077252672 (+1 ms)Cleaning up temporary data from old regions at 1676077252765 (+93 ms)Cleaning up detritus from prior splits at 1676077252766 (+1 ms)Region opened successfully at 1676077252779 (+13 ms) 2023-02-11 01:00:52,781 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2023-02-11 01:00:52,800 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=5 2023-02-11 01:00:52,800 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(562): Starting 5 core workers (bigger of cpus/4 or 16) with max (burst) worker count=50 2023-02-11 01:00:52,802 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-02-11 01:00:52,803 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-02-11 01:00:52,834 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 30 msec 2023-02-11 01:00:52,834 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(95): Instantiated, coreThreads=3 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2023-02-11 01:00:52,858 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(253): hbase:meta replica znodes: [] 2023-02-11 01:00:52,863 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 0 possibly 'live' servers, and 0 'splitting'. 2023-02-11 01:00:52,888 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-02-11 01:00:52,891 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, ServerLocalityCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-02-11 01:00:52,892 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:33145-0x10040e51f9c0000, quorum=127.0.0.1:63347, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2023-02-11 01:00:52,897 INFO [master/jenkins-hbase4:0:becomeActiveMaster] normalizer.RegionNormalizerWorker(118): Normalizer rate limit set to unlimited 2023-02-11 01:00:52,901 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:33145-0x10040e51f9c0000, quorum=127.0.0.1:63347, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2023-02-11 01:00:52,903 DEBUG [Listener at localhost/46601-EventThread] zookeeper.ZKWatcher(600): master:33145-0x10040e51f9c0000, quorum=127.0.0.1:63347, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-11 01:00:52,905 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:33145-0x10040e51f9c0000, quorum=127.0.0.1:63347, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2023-02-11 01:00:52,905 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:33145-0x10040e51f9c0000, quorum=127.0.0.1:63347, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2023-02-11 01:00:52,917 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:33145-0x10040e51f9c0000, quorum=127.0.0.1:63347, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/snapshot-cleanup 2023-02-11 01:00:52,921 DEBUG [Listener at localhost/46601-EventThread] zookeeper.ZKWatcher(600): regionserver:42073-0x10040e51f9c0003, quorum=127.0.0.1:63347, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-02-11 01:00:52,921 DEBUG [Listener at localhost/46601-EventThread] zookeeper.ZKWatcher(600): master:33145-0x10040e51f9c0000, quorum=127.0.0.1:63347, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-02-11 01:00:52,921 DEBUG [Listener at localhost/46601-EventThread] zookeeper.ZKWatcher(600): regionserver:44999-0x10040e51f9c0002, quorum=127.0.0.1:63347, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-02-11 01:00:52,921 DEBUG [Listener at localhost/46601-EventThread] zookeeper.ZKWatcher(600): regionserver:41571-0x10040e51f9c0001, quorum=127.0.0.1:63347, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-02-11 01:00:52,921 DEBUG [Listener at localhost/46601-EventThread] zookeeper.ZKWatcher(600): master:33145-0x10040e51f9c0000, quorum=127.0.0.1:63347, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-11 01:00:52,922 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(739): Active/primary master=jenkins-hbase4.apache.org,33145,1676077250844, sessionid=0x10040e51f9c0000, setting cluster-up flag (Was=false) 2023-02-11 01:00:52,938 DEBUG [Listener at localhost/46601-EventThread] zookeeper.ZKWatcher(600): master:33145-0x10040e51f9c0000, quorum=127.0.0.1:63347, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-11 01:00:52,942 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2023-02-11 01:00:52,943 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,33145,1676077250844 2023-02-11 01:00:52,949 DEBUG [Listener at localhost/46601-EventThread] zookeeper.ZKWatcher(600): master:33145-0x10040e51f9c0000, quorum=127.0.0.1:63347, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-11 01:00:52,955 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2023-02-11 01:00:52,956 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,33145,1676077250844 2023-02-11 01:00:52,958 WARN [master/jenkins-hbase4:0:becomeActiveMaster] snapshot.SnapshotManager(302): Couldn't delete working snapshot directory: hdfs://localhost:44295/user/jenkins/test-data/593564ae-939c-6914-503a-94843f3f7058/.hbase-snapshot/.tmp 2023-02-11 01:00:52,988 INFO [RS:0;jenkins-hbase4:41571] regionserver.HRegionServer(952): ClusterId : eb576bd2-5fcf-4742-a837-61e802658a82 2023-02-11 01:00:52,988 INFO [RS:2;jenkins-hbase4:42073] regionserver.HRegionServer(952): ClusterId : eb576bd2-5fcf-4742-a837-61e802658a82 2023-02-11 01:00:52,988 INFO [RS:1;jenkins-hbase4:44999] regionserver.HRegionServer(952): ClusterId : eb576bd2-5fcf-4742-a837-61e802658a82 2023-02-11 01:00:52,993 DEBUG [RS:1;jenkins-hbase4:44999] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-02-11 01:00:52,993 DEBUG [RS:0;jenkins-hbase4:41571] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-02-11 01:00:52,993 DEBUG [RS:2;jenkins-hbase4:42073] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-02-11 01:00:52,999 DEBUG [RS:2;jenkins-hbase4:42073] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-02-11 01:00:52,999 DEBUG [RS:0;jenkins-hbase4:41571] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-02-11 01:00:52,999 DEBUG [RS:2;jenkins-hbase4:42073] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-02-11 01:00:52,999 DEBUG [RS:0;jenkins-hbase4:41571] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-02-11 01:00:53,002 DEBUG [RS:1;jenkins-hbase4:44999] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-02-11 01:00:53,002 DEBUG [RS:1;jenkins-hbase4:44999] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-02-11 01:00:53,005 DEBUG [RS:2;jenkins-hbase4:42073] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-02-11 01:00:53,005 DEBUG [RS:0;jenkins-hbase4:41571] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-02-11 01:00:53,005 DEBUG [RS:1;jenkins-hbase4:44999] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-02-11 01:00:53,009 DEBUG [RS:2;jenkins-hbase4:42073] zookeeper.ReadOnlyZKClient(139): Connect 0x669fb05d to 127.0.0.1:63347 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-11 01:00:53,010 DEBUG [RS:1;jenkins-hbase4:44999] zookeeper.ReadOnlyZKClient(139): Connect 0x2a3b18e1 to 127.0.0.1:63347 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-11 01:00:53,010 DEBUG [RS:0;jenkins-hbase4:41571] zookeeper.ReadOnlyZKClient(139): Connect 0x0d01c485 to 127.0.0.1:63347 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-11 01:00:53,018 DEBUG [RS:0;jenkins-hbase4:41571] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3c62b55, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-11 01:00:53,019 DEBUG [RS:2;jenkins-hbase4:42073] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@685842fe, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-11 01:00:53,019 DEBUG [RS:0;jenkins-hbase4:41571] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7f9aff5, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-02-11 01:00:53,019 DEBUG [RS:2;jenkins-hbase4:42073] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6c67db2d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-02-11 01:00:53,020 DEBUG [RS:1;jenkins-hbase4:44999] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@24fce3d6, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-11 01:00:53,020 DEBUG [RS:1;jenkins-hbase4:44999] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@20c37391, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-02-11 01:00:53,043 DEBUG [RS:0;jenkins-hbase4:41571] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:0;jenkins-hbase4:41571 2023-02-11 01:00:53,044 DEBUG [RS:2;jenkins-hbase4:42073] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:2;jenkins-hbase4:42073 2023-02-11 01:00:53,047 DEBUG [RS:1;jenkins-hbase4:44999] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:1;jenkins-hbase4:44999 2023-02-11 01:00:53,047 INFO [RS:0;jenkins-hbase4:41571] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-02-11 01:00:53,048 INFO [RS:1;jenkins-hbase4:44999] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-02-11 01:00:53,048 INFO [RS:0;jenkins-hbase4:41571] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-02-11 01:00:53,047 INFO [RS:2;jenkins-hbase4:42073] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-02-11 01:00:53,048 DEBUG [RS:0;jenkins-hbase4:41571] regionserver.HRegionServer(1023): About to register with Master. 2023-02-11 01:00:53,048 INFO [RS:1;jenkins-hbase4:44999] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-02-11 01:00:53,048 INFO [RS:2;jenkins-hbase4:42073] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-02-11 01:00:53,048 DEBUG [RS:1;jenkins-hbase4:44999] regionserver.HRegionServer(1023): About to register with Master. 2023-02-11 01:00:53,049 DEBUG [RS:2;jenkins-hbase4:42073] regionserver.HRegionServer(1023): About to register with Master. 2023-02-11 01:00:53,051 INFO [RS:1;jenkins-hbase4:44999] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,33145,1676077250844 with isa=jenkins-hbase4.apache.org/172.31.14.131:44999, startcode=1676077252037 2023-02-11 01:00:53,051 INFO [RS:0;jenkins-hbase4:41571] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,33145,1676077250844 with isa=jenkins-hbase4.apache.org/172.31.14.131:41571, startcode=1676077251993 2023-02-11 01:00:53,051 INFO [RS:2;jenkins-hbase4:42073] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,33145,1676077250844 with isa=jenkins-hbase4.apache.org/172.31.14.131:42073, startcode=1676077252069 2023-02-11 01:00:53,069 DEBUG [RS:1;jenkins-hbase4:44999] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-02-11 01:00:53,069 DEBUG [RS:2;jenkins-hbase4:42073] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-02-11 01:00:53,069 DEBUG [RS:0;jenkins-hbase4:41571] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-02-11 01:00:53,088 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT; InitMetaProcedure table=hbase:meta 2023-02-11 01:00:53,105 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-02-11 01:00:53,105 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-02-11 01:00:53,105 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-02-11 01:00:53,105 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-02-11 01:00:53,105 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-master/jenkins-hbase4:0, corePoolSize=10, maxPoolSize=10 2023-02-11 01:00:53,105 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 01:00:53,105 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_MERGE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-02-11 01:00:53,105 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 01:00:53,107 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1676077283107 2023-02-11 01:00:53,110 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2023-02-11 01:00:53,112 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT, locked=true; InitMetaProcedure table=hbase:meta 2023-02-11 01:00:53,113 INFO [PEWorker-1] procedure.InitMetaProcedure(71): BOOTSTRAP: creating hbase:meta region 2023-02-11 01:00:53,120 INFO [PEWorker-1] util.FSTableDescriptors(128): Creating new hbase:meta table descriptor 'hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-02-11 01:00:53,122 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2023-02-11 01:00:53,126 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:37361, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins.hfs.0 (auth:SIMPLE), service=RegionServerStatusService 2023-02-11 01:00:53,126 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:52977, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins.hfs.2 (auth:SIMPLE), service=RegionServerStatusService 2023-02-11 01:00:53,126 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:49299, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins.hfs.1 (auth:SIMPLE), service=RegionServerStatusService 2023-02-11 01:00:53,130 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2023-02-11 01:00:53,131 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2023-02-11 01:00:53,131 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2023-02-11 01:00:53,132 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2023-02-11 01:00:53,134 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-02-11 01:00:53,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33145] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2815) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerStartup(MasterRpcServices.java:577) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15952) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 01:00:53,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33145] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2815) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerStartup(MasterRpcServices.java:577) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15952) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 01:00:53,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33145] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2815) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerStartup(MasterRpcServices.java:577) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15952) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-11 01:00:53,137 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2023-02-11 01:00:53,139 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2023-02-11 01:00:53,139 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2023-02-11 01:00:53,142 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2023-02-11 01:00:53,142 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2023-02-11 01:00:53,144 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(242): Starting for large file=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1676077253144,5,FailOnTimeoutGroup] 2023-02-11 01:00:53,145 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(257): Starting for small files=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1676077253144,5,FailOnTimeoutGroup] 2023-02-11 01:00:53,145 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-02-11 01:00:53,145 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1451): Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2023-02-11 01:00:53,146 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2023-02-11 01:00:53,147 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2023-02-11 01:00:53,170 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:44295/user/jenkins/test-data/593564ae-939c-6914-503a-94843f3f7058/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-02-11 01:00:53,171 INFO [PEWorker-1] util.FSTableDescriptors(135): Updated hbase:meta table descriptor to hdfs://localhost:44295/user/jenkins/test-data/593564ae-939c-6914-503a-94843f3f7058/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-02-11 01:00:53,171 INFO [PEWorker-1] regionserver.HRegion(7671): creating {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:44295/user/jenkins/test-data/593564ae-939c-6914-503a-94843f3f7058 2023-02-11 01:00:53,176 DEBUG [RS:2;jenkins-hbase4:42073] regionserver.HRegionServer(2829): Master is not running yet 2023-02-11 01:00:53,176 DEBUG [RS:1;jenkins-hbase4:44999] regionserver.HRegionServer(2829): Master is not running yet 2023-02-11 01:00:53,177 WARN [RS:2;jenkins-hbase4:42073] regionserver.HRegionServer(1031): reportForDuty failed; sleeping 100 ms and then retrying. 2023-02-11 01:00:53,176 DEBUG [RS:0;jenkins-hbase4:41571] regionserver.HRegionServer(2829): Master is not running yet 2023-02-11 01:00:53,177 WARN [RS:1;jenkins-hbase4:44999] regionserver.HRegionServer(1031): reportForDuty failed; sleeping 100 ms and then retrying. 2023-02-11 01:00:53,177 WARN [RS:0;jenkins-hbase4:41571] regionserver.HRegionServer(1031): reportForDuty failed; sleeping 100 ms and then retrying. 2023-02-11 01:00:53,195 DEBUG [PEWorker-1] regionserver.HRegion(865): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 01:00:53,197 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-02-11 01:00:53,199 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44295/user/jenkins/test-data/593564ae-939c-6914-503a-94843f3f7058/data/hbase/meta/1588230740/info 2023-02-11 01:00:53,200 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-02-11 01:00:53,201 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 01:00:53,201 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-02-11 01:00:53,203 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44295/user/jenkins/test-data/593564ae-939c-6914-503a-94843f3f7058/data/hbase/meta/1588230740/rep_barrier 2023-02-11 01:00:53,204 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-02-11 01:00:53,205 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 01:00:53,205 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-02-11 01:00:53,207 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44295/user/jenkins/test-data/593564ae-939c-6914-503a-94843f3f7058/data/hbase/meta/1588230740/table 2023-02-11 01:00:53,207 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-02-11 01:00:53,208 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 01:00:53,210 DEBUG [PEWorker-1] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:44295/user/jenkins/test-data/593564ae-939c-6914-503a-94843f3f7058/data/hbase/meta/1588230740 2023-02-11 01:00:53,210 DEBUG [PEWorker-1] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:44295/user/jenkins/test-data/593564ae-939c-6914-503a-94843f3f7058/data/hbase/meta/1588230740 2023-02-11 01:00:53,214 DEBUG [PEWorker-1] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-02-11 01:00:53,216 DEBUG [PEWorker-1] regionserver.HRegion(1054): writing seq id for 1588230740 2023-02-11 01:00:53,219 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44295/user/jenkins/test-data/593564ae-939c-6914-503a-94843f3f7058/data/hbase/meta/1588230740/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-11 01:00:53,220 INFO [PEWorker-1] regionserver.HRegion(1071): Opened 1588230740; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=67052713, jitterRate=-8.367151021957397E-4}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-02-11 01:00:53,220 DEBUG [PEWorker-1] regionserver.HRegion(964): Region open journal for 1588230740: Writing region info on filesystem at 1676077253195Initializing all the Stores at 1676077253197 (+2 ms)Instantiating store for column family {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} at 1676077253197Instantiating store for column family {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676077253197Instantiating store for column family {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} at 1676077253197Cleaning up temporary data from old regions at 1676077253212 (+15 ms)Cleaning up detritus from prior splits at 1676077253213 (+1 ms)Region opened successfully at 1676077253220 (+7 ms) 2023-02-11 01:00:53,220 DEBUG [PEWorker-1] regionserver.HRegion(1603): Closing 1588230740, disabling compactions & flushes 2023-02-11 01:00:53,220 INFO [PEWorker-1] regionserver.HRegion(1625): Closing region hbase:meta,,1.1588230740 2023-02-11 01:00:53,220 DEBUG [PEWorker-1] regionserver.HRegion(1646): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-02-11 01:00:53,220 DEBUG [PEWorker-1] regionserver.HRegion(1713): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-02-11 01:00:53,220 DEBUG [PEWorker-1] regionserver.HRegion(1723): Updates disabled for region hbase:meta,,1.1588230740 2023-02-11 01:00:53,221 INFO [PEWorker-1] regionserver.HRegion(1837): Closed hbase:meta,,1.1588230740 2023-02-11 01:00:53,222 DEBUG [PEWorker-1] regionserver.HRegion(1557): Region close journal for 1588230740: Waiting for close lock at 1676077253220Disabling compacts and flushes for region at 1676077253220Disabling writes for close at 1676077253220Writing region close event to WAL at 1676077253221 (+1 ms)Closed at 1676077253221 2023-02-11 01:00:53,227 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_ASSIGN_META, locked=true; InitMetaProcedure table=hbase:meta 2023-02-11 01:00:53,227 INFO [PEWorker-1] procedure.InitMetaProcedure(103): Going to assign meta 2023-02-11 01:00:53,236 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-02-11 01:00:53,247 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-02-11 01:00:53,250 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OFFLINE, location=null; forceNewPlan=false, retain=false 2023-02-11 01:00:53,278 INFO [RS:2;jenkins-hbase4:42073] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,33145,1676077250844 with isa=jenkins-hbase4.apache.org/172.31.14.131:42073, startcode=1676077252069 2023-02-11 01:00:53,278 INFO [RS:0;jenkins-hbase4:41571] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,33145,1676077250844 with isa=jenkins-hbase4.apache.org/172.31.14.131:41571, startcode=1676077251993 2023-02-11 01:00:53,278 INFO [RS:1;jenkins-hbase4:44999] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,33145,1676077250844 with isa=jenkins-hbase4.apache.org/172.31.14.131:44999, startcode=1676077252037 2023-02-11 01:00:53,284 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=33145] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,44999,1676077252037 2023-02-11 01:00:53,284 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33145] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,41571,1676077251993 2023-02-11 01:00:53,285 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=33145] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,42073,1676077252069 2023-02-11 01:00:53,291 DEBUG [RS:2;jenkins-hbase4:42073] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost:44295/user/jenkins/test-data/593564ae-939c-6914-503a-94843f3f7058 2023-02-11 01:00:53,291 DEBUG [RS:1;jenkins-hbase4:44999] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost:44295/user/jenkins/test-data/593564ae-939c-6914-503a-94843f3f7058 2023-02-11 01:00:53,291 DEBUG [RS:0;jenkins-hbase4:41571] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost:44295/user/jenkins/test-data/593564ae-939c-6914-503a-94843f3f7058 2023-02-11 01:00:53,292 DEBUG [RS:1;jenkins-hbase4:44999] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost:44295 2023-02-11 01:00:53,291 DEBUG [RS:2;jenkins-hbase4:42073] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost:44295 2023-02-11 01:00:53,292 DEBUG [RS:1;jenkins-hbase4:44999] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-02-11 01:00:53,292 DEBUG [RS:0;jenkins-hbase4:41571] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost:44295 2023-02-11 01:00:53,292 DEBUG [RS:2;jenkins-hbase4:42073] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-02-11 01:00:53,292 DEBUG [RS:0;jenkins-hbase4:41571] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-02-11 01:00:53,300 DEBUG [Listener at localhost/46601-EventThread] zookeeper.ZKWatcher(600): master:33145-0x10040e51f9c0000, quorum=127.0.0.1:63347, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-11 01:00:53,301 DEBUG [RS:2;jenkins-hbase4:42073] zookeeper.ZKUtil(162): regionserver:42073-0x10040e51f9c0003, quorum=127.0.0.1:63347, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,42073,1676077252069 2023-02-11 01:00:53,301 DEBUG [RS:1;jenkins-hbase4:44999] zookeeper.ZKUtil(162): regionserver:44999-0x10040e51f9c0002, quorum=127.0.0.1:63347, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,44999,1676077252037 2023-02-11 01:00:53,301 DEBUG [RS:0;jenkins-hbase4:41571] zookeeper.ZKUtil(162): regionserver:41571-0x10040e51f9c0001, quorum=127.0.0.1:63347, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,41571,1676077251993 2023-02-11 01:00:53,301 WARN [RS:1;jenkins-hbase4:44999] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-02-11 01:00:53,301 WARN [RS:2;jenkins-hbase4:42073] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-02-11 01:00:53,302 INFO [RS:1;jenkins-hbase4:44999] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-11 01:00:53,302 INFO [RS:2;jenkins-hbase4:42073] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-11 01:00:53,302 WARN [RS:0;jenkins-hbase4:41571] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-02-11 01:00:53,302 DEBUG [RS:1;jenkins-hbase4:44999] regionserver.HRegionServer(1947): logDir=hdfs://localhost:44295/user/jenkins/test-data/593564ae-939c-6914-503a-94843f3f7058/WALs/jenkins-hbase4.apache.org,44999,1676077252037 2023-02-11 01:00:53,303 INFO [RS:0;jenkins-hbase4:41571] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-11 01:00:53,303 DEBUG [RS:2;jenkins-hbase4:42073] regionserver.HRegionServer(1947): logDir=hdfs://localhost:44295/user/jenkins/test-data/593564ae-939c-6914-503a-94843f3f7058/WALs/jenkins-hbase4.apache.org,42073,1676077252069 2023-02-11 01:00:53,303 DEBUG [RS:0;jenkins-hbase4:41571] regionserver.HRegionServer(1947): logDir=hdfs://localhost:44295/user/jenkins/test-data/593564ae-939c-6914-503a-94843f3f7058/WALs/jenkins-hbase4.apache.org,41571,1676077251993 2023-02-11 01:00:53,303 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,44999,1676077252037] 2023-02-11 01:00:53,303 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,42073,1676077252069] 2023-02-11 01:00:53,303 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,41571,1676077251993] 2023-02-11 01:00:53,319 DEBUG [RS:0;jenkins-hbase4:41571] zookeeper.ZKUtil(162): regionserver:41571-0x10040e51f9c0001, quorum=127.0.0.1:63347, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,42073,1676077252069 2023-02-11 01:00:53,319 DEBUG [RS:2;jenkins-hbase4:42073] zookeeper.ZKUtil(162): regionserver:42073-0x10040e51f9c0003, quorum=127.0.0.1:63347, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,42073,1676077252069 2023-02-11 01:00:53,319 DEBUG [RS:1;jenkins-hbase4:44999] zookeeper.ZKUtil(162): regionserver:44999-0x10040e51f9c0002, quorum=127.0.0.1:63347, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,42073,1676077252069 2023-02-11 01:00:53,320 DEBUG [RS:0;jenkins-hbase4:41571] zookeeper.ZKUtil(162): regionserver:41571-0x10040e51f9c0001, quorum=127.0.0.1:63347, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,41571,1676077251993 2023-02-11 01:00:53,320 DEBUG [RS:2;jenkins-hbase4:42073] zookeeper.ZKUtil(162): regionserver:42073-0x10040e51f9c0003, quorum=127.0.0.1:63347, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,41571,1676077251993 2023-02-11 01:00:53,320 DEBUG [RS:1;jenkins-hbase4:44999] zookeeper.ZKUtil(162): regionserver:44999-0x10040e51f9c0002, quorum=127.0.0.1:63347, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,41571,1676077251993 2023-02-11 01:00:53,320 DEBUG [RS:0;jenkins-hbase4:41571] zookeeper.ZKUtil(162): regionserver:41571-0x10040e51f9c0001, quorum=127.0.0.1:63347, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,44999,1676077252037 2023-02-11 01:00:53,320 DEBUG [RS:2;jenkins-hbase4:42073] zookeeper.ZKUtil(162): regionserver:42073-0x10040e51f9c0003, quorum=127.0.0.1:63347, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,44999,1676077252037 2023-02-11 01:00:53,320 DEBUG [RS:1;jenkins-hbase4:44999] zookeeper.ZKUtil(162): regionserver:44999-0x10040e51f9c0002, quorum=127.0.0.1:63347, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,44999,1676077252037 2023-02-11 01:00:53,330 DEBUG [RS:0;jenkins-hbase4:41571] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-02-11 01:00:53,330 DEBUG [RS:2;jenkins-hbase4:42073] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-02-11 01:00:53,330 DEBUG [RS:1;jenkins-hbase4:44999] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-02-11 01:00:53,339 INFO [RS:0;jenkins-hbase4:41571] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-02-11 01:00:53,339 INFO [RS:1;jenkins-hbase4:44999] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-02-11 01:00:53,339 INFO [RS:2;jenkins-hbase4:42073] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-02-11 01:00:53,359 INFO [RS:0;jenkins-hbase4:41571] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-02-11 01:00:53,359 INFO [RS:1;jenkins-hbase4:44999] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-02-11 01:00:53,359 INFO [RS:2;jenkins-hbase4:42073] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-02-11 01:00:53,364 INFO [RS:2;jenkins-hbase4:42073] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-02-11 01:00:53,364 INFO [RS:1;jenkins-hbase4:44999] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-02-11 01:00:53,364 INFO [RS:2;jenkins-hbase4:42073] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-02-11 01:00:53,364 INFO [RS:0;jenkins-hbase4:41571] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-02-11 01:00:53,364 INFO [RS:1;jenkins-hbase4:44999] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-02-11 01:00:53,364 INFO [RS:0;jenkins-hbase4:41571] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-02-11 01:00:53,365 INFO [RS:2;jenkins-hbase4:42073] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-02-11 01:00:53,365 INFO [RS:0;jenkins-hbase4:41571] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-02-11 01:00:53,365 INFO [RS:1;jenkins-hbase4:44999] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-02-11 01:00:53,373 INFO [RS:0;jenkins-hbase4:41571] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-02-11 01:00:53,373 INFO [RS:2;jenkins-hbase4:42073] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-02-11 01:00:53,373 INFO [RS:1;jenkins-hbase4:44999] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-02-11 01:00:53,373 DEBUG [RS:0;jenkins-hbase4:41571] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 01:00:53,373 DEBUG [RS:1;jenkins-hbase4:44999] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 01:00:53,373 DEBUG [RS:0;jenkins-hbase4:41571] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 01:00:53,373 DEBUG [RS:1;jenkins-hbase4:44999] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 01:00:53,373 DEBUG [RS:0;jenkins-hbase4:41571] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 01:00:53,373 DEBUG [RS:1;jenkins-hbase4:44999] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 01:00:53,373 DEBUG [RS:0;jenkins-hbase4:41571] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 01:00:53,373 DEBUG [RS:1;jenkins-hbase4:44999] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 01:00:53,374 DEBUG [RS:0;jenkins-hbase4:41571] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 01:00:53,374 DEBUG [RS:1;jenkins-hbase4:44999] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 01:00:53,374 DEBUG [RS:0;jenkins-hbase4:41571] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-02-11 01:00:53,374 DEBUG [RS:1;jenkins-hbase4:44999] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-02-11 01:00:53,374 DEBUG [RS:0;jenkins-hbase4:41571] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 01:00:53,374 DEBUG [RS:1;jenkins-hbase4:44999] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 01:00:53,374 DEBUG [RS:0;jenkins-hbase4:41571] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 01:00:53,374 DEBUG [RS:1;jenkins-hbase4:44999] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 01:00:53,374 DEBUG [RS:0;jenkins-hbase4:41571] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 01:00:53,374 DEBUG [RS:1;jenkins-hbase4:44999] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 01:00:53,373 DEBUG [RS:2;jenkins-hbase4:42073] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 01:00:53,374 DEBUG [RS:1;jenkins-hbase4:44999] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 01:00:53,374 DEBUG [RS:0;jenkins-hbase4:41571] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 01:00:53,374 DEBUG [RS:2;jenkins-hbase4:42073] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 01:00:53,374 DEBUG [RS:2;jenkins-hbase4:42073] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 01:00:53,375 DEBUG [RS:2;jenkins-hbase4:42073] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 01:00:53,375 DEBUG [RS:2;jenkins-hbase4:42073] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 01:00:53,376 DEBUG [RS:2;jenkins-hbase4:42073] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-02-11 01:00:53,376 DEBUG [RS:2;jenkins-hbase4:42073] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 01:00:53,376 DEBUG [RS:2;jenkins-hbase4:42073] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 01:00:53,376 INFO [RS:0;jenkins-hbase4:41571] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-02-11 01:00:53,376 DEBUG [RS:2;jenkins-hbase4:42073] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 01:00:53,376 INFO [RS:0;jenkins-hbase4:41571] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-02-11 01:00:53,376 DEBUG [RS:2;jenkins-hbase4:42073] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 01:00:53,376 INFO [RS:0;jenkins-hbase4:41571] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-02-11 01:00:53,379 INFO [RS:1;jenkins-hbase4:44999] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-02-11 01:00:53,379 INFO [RS:1;jenkins-hbase4:44999] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-02-11 01:00:53,379 INFO [RS:1;jenkins-hbase4:44999] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-02-11 01:00:53,380 INFO [RS:2;jenkins-hbase4:42073] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-02-11 01:00:53,380 INFO [RS:2;jenkins-hbase4:42073] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-02-11 01:00:53,380 INFO [RS:2;jenkins-hbase4:42073] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-02-11 01:00:53,397 INFO [RS:1;jenkins-hbase4:44999] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-02-11 01:00:53,397 INFO [RS:2;jenkins-hbase4:42073] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-02-11 01:00:53,397 INFO [RS:0;jenkins-hbase4:41571] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-02-11 01:00:53,400 INFO [RS:1;jenkins-hbase4:44999] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,44999,1676077252037-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-02-11 01:00:53,400 INFO [RS:2;jenkins-hbase4:42073] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,42073,1676077252069-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-02-11 01:00:53,400 INFO [RS:0;jenkins-hbase4:41571] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,41571,1676077251993-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-02-11 01:00:53,402 DEBUG [jenkins-hbase4:33145] assignment.AssignmentManager(2178): Processing assignQueue; systemServersCount=3, allServersCount=3 2023-02-11 01:00:53,406 DEBUG [jenkins-hbase4:33145] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-02-11 01:00:53,413 DEBUG [jenkins-hbase4:33145] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-11 01:00:53,413 DEBUG [jenkins-hbase4:33145] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-11 01:00:53,413 DEBUG [jenkins-hbase4:33145] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-11 01:00:53,413 DEBUG [jenkins-hbase4:33145] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-11 01:00:53,417 INFO [PEWorker-3] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,42073,1676077252069, state=OPENING 2023-02-11 01:00:53,423 INFO [RS:2;jenkins-hbase4:42073] regionserver.Replication(203): jenkins-hbase4.apache.org,42073,1676077252069 started 2023-02-11 01:00:53,423 INFO [RS:1;jenkins-hbase4:44999] regionserver.Replication(203): jenkins-hbase4.apache.org,44999,1676077252037 started 2023-02-11 01:00:53,423 INFO [RS:2;jenkins-hbase4:42073] regionserver.HRegionServer(1638): Serving as jenkins-hbase4.apache.org,42073,1676077252069, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:42073, sessionid=0x10040e51f9c0003 2023-02-11 01:00:53,423 INFO [RS:1;jenkins-hbase4:44999] regionserver.HRegionServer(1638): Serving as jenkins-hbase4.apache.org,44999,1676077252037, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:44999, sessionid=0x10040e51f9c0002 2023-02-11 01:00:53,423 DEBUG [RS:1;jenkins-hbase4:44999] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-02-11 01:00:53,423 DEBUG [RS:2;jenkins-hbase4:42073] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-02-11 01:00:53,423 DEBUG [RS:1;jenkins-hbase4:44999] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,44999,1676077252037 2023-02-11 01:00:53,423 DEBUG [RS:2;jenkins-hbase4:42073] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,42073,1676077252069 2023-02-11 01:00:53,424 DEBUG [RS:1;jenkins-hbase4:44999] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,44999,1676077252037' 2023-02-11 01:00:53,424 DEBUG [RS:2;jenkins-hbase4:42073] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,42073,1676077252069' 2023-02-11 01:00:53,424 DEBUG [RS:2;jenkins-hbase4:42073] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-11 01:00:53,424 DEBUG [RS:1;jenkins-hbase4:44999] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-11 01:00:53,425 DEBUG [RS:1;jenkins-hbase4:44999] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-11 01:00:53,425 DEBUG [RS:2;jenkins-hbase4:42073] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-11 01:00:53,426 INFO [RS:0;jenkins-hbase4:41571] regionserver.Replication(203): jenkins-hbase4.apache.org,41571,1676077251993 started 2023-02-11 01:00:53,426 INFO [RS:0;jenkins-hbase4:41571] regionserver.HRegionServer(1638): Serving as jenkins-hbase4.apache.org,41571,1676077251993, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:41571, sessionid=0x10040e51f9c0001 2023-02-11 01:00:53,426 DEBUG [RS:1;jenkins-hbase4:44999] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-02-11 01:00:53,426 DEBUG [RS:0;jenkins-hbase4:41571] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-02-11 01:00:53,426 DEBUG [RS:2;jenkins-hbase4:42073] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-02-11 01:00:53,426 DEBUG [RS:0;jenkins-hbase4:41571] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,41571,1676077251993 2023-02-11 01:00:53,426 DEBUG [RS:1;jenkins-hbase4:44999] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-02-11 01:00:53,426 DEBUG [RS:0;jenkins-hbase4:41571] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,41571,1676077251993' 2023-02-11 01:00:53,426 DEBUG [RS:2;jenkins-hbase4:42073] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-02-11 01:00:53,427 DEBUG [PEWorker-3] zookeeper.MetaTableLocator(240): hbase:meta region location doesn't exist, create it 2023-02-11 01:00:53,427 DEBUG [RS:2;jenkins-hbase4:42073] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,42073,1676077252069 2023-02-11 01:00:53,426 DEBUG [RS:0;jenkins-hbase4:41571] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-11 01:00:53,426 DEBUG [RS:1;jenkins-hbase4:44999] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,44999,1676077252037 2023-02-11 01:00:53,427 DEBUG [RS:2;jenkins-hbase4:42073] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,42073,1676077252069' 2023-02-11 01:00:53,427 DEBUG [RS:1;jenkins-hbase4:44999] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,44999,1676077252037' 2023-02-11 01:00:53,427 DEBUG [RS:1;jenkins-hbase4:44999] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-02-11 01:00:53,427 DEBUG [RS:2;jenkins-hbase4:42073] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-02-11 01:00:53,428 DEBUG [Listener at localhost/46601-EventThread] zookeeper.ZKWatcher(600): master:33145-0x10040e51f9c0000, quorum=127.0.0.1:63347, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-11 01:00:53,429 DEBUG [RS:2;jenkins-hbase4:42073] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-02-11 01:00:53,429 DEBUG [RS:0;jenkins-hbase4:41571] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-11 01:00:53,429 DEBUG [RS:1;jenkins-hbase4:44999] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-02-11 01:00:53,429 DEBUG [RS:0;jenkins-hbase4:41571] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-02-11 01:00:53,429 DEBUG [RS:2;jenkins-hbase4:42073] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-02-11 01:00:53,429 DEBUG [RS:0;jenkins-hbase4:41571] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-02-11 01:00:53,429 DEBUG [RS:0;jenkins-hbase4:41571] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,41571,1676077251993 2023-02-11 01:00:53,429 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-02-11 01:00:53,430 DEBUG [RS:0;jenkins-hbase4:41571] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,41571,1676077251993' 2023-02-11 01:00:53,430 DEBUG [RS:0;jenkins-hbase4:41571] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-02-11 01:00:53,430 DEBUG [RS:1;jenkins-hbase4:44999] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-02-11 01:00:53,429 INFO [RS:2;jenkins-hbase4:42073] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-02-11 01:00:53,430 INFO [RS:1;jenkins-hbase4:44999] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-02-11 01:00:53,430 INFO [RS:1;jenkins-hbase4:44999] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-02-11 01:00:53,430 INFO [RS:2;jenkins-hbase4:42073] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-02-11 01:00:53,431 DEBUG [RS:0;jenkins-hbase4:41571] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-02-11 01:00:53,431 DEBUG [RS:0;jenkins-hbase4:41571] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-02-11 01:00:53,431 INFO [RS:0;jenkins-hbase4:41571] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-02-11 01:00:53,431 INFO [RS:0;jenkins-hbase4:41571] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-02-11 01:00:53,433 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=3, ppid=2, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,42073,1676077252069}] 2023-02-11 01:00:53,541 INFO [RS:0;jenkins-hbase4:41571] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C41571%2C1676077251993, suffix=, logDir=hdfs://localhost:44295/user/jenkins/test-data/593564ae-939c-6914-503a-94843f3f7058/WALs/jenkins-hbase4.apache.org,41571,1676077251993, archiveDir=hdfs://localhost:44295/user/jenkins/test-data/593564ae-939c-6914-503a-94843f3f7058/oldWALs, maxLogs=32 2023-02-11 01:00:53,541 INFO [RS:2;jenkins-hbase4:42073] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C42073%2C1676077252069, suffix=, logDir=hdfs://localhost:44295/user/jenkins/test-data/593564ae-939c-6914-503a-94843f3f7058/WALs/jenkins-hbase4.apache.org,42073,1676077252069, archiveDir=hdfs://localhost:44295/user/jenkins/test-data/593564ae-939c-6914-503a-94843f3f7058/oldWALs, maxLogs=32 2023-02-11 01:00:53,541 INFO [RS:1;jenkins-hbase4:44999] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C44999%2C1676077252037, suffix=, logDir=hdfs://localhost:44295/user/jenkins/test-data/593564ae-939c-6914-503a-94843f3f7058/WALs/jenkins-hbase4.apache.org,44999,1676077252037, archiveDir=hdfs://localhost:44295/user/jenkins/test-data/593564ae-939c-6914-503a-94843f3f7058/oldWALs, maxLogs=32 2023-02-11 01:00:53,565 DEBUG [RS-EventLoopGroup-5-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:39915,DS-9ebc61aa-9b9c-41d9-abc9-b4b9b6de6337,DISK] 2023-02-11 01:00:53,565 DEBUG [RS-EventLoopGroup-5-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:39157,DS-8f401005-340e-4621-a18e-f461eb4d4ab2,DISK] 2023-02-11 01:00:53,566 DEBUG [RS-EventLoopGroup-5-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:37515,DS-19365098-07a0-4163-8732-27153e4b255b,DISK] 2023-02-11 01:00:53,577 INFO [RS:0;jenkins-hbase4:41571] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/593564ae-939c-6914-503a-94843f3f7058/WALs/jenkins-hbase4.apache.org,41571,1676077251993/jenkins-hbase4.apache.org%2C41571%2C1676077251993.1676077253545 2023-02-11 01:00:53,583 DEBUG [RS:0;jenkins-hbase4:41571] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:39915,DS-9ebc61aa-9b9c-41d9-abc9-b4b9b6de6337,DISK], DatanodeInfoWithStorage[127.0.0.1:39157,DS-8f401005-340e-4621-a18e-f461eb4d4ab2,DISK], DatanodeInfoWithStorage[127.0.0.1:37515,DS-19365098-07a0-4163-8732-27153e4b255b,DISK]] 2023-02-11 01:00:53,585 DEBUG [RS-EventLoopGroup-5-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:37515,DS-19365098-07a0-4163-8732-27153e4b255b,DISK] 2023-02-11 01:00:53,586 DEBUG [RS-EventLoopGroup-5-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:39915,DS-9ebc61aa-9b9c-41d9-abc9-b4b9b6de6337,DISK] 2023-02-11 01:00:53,586 DEBUG [RS-EventLoopGroup-5-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:39157,DS-8f401005-340e-4621-a18e-f461eb4d4ab2,DISK] 2023-02-11 01:00:53,586 DEBUG [RS-EventLoopGroup-5-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:39915,DS-9ebc61aa-9b9c-41d9-abc9-b4b9b6de6337,DISK] 2023-02-11 01:00:53,595 DEBUG [RS-EventLoopGroup-5-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:39157,DS-8f401005-340e-4621-a18e-f461eb4d4ab2,DISK] 2023-02-11 01:00:53,596 DEBUG [RS-EventLoopGroup-5-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:37515,DS-19365098-07a0-4163-8732-27153e4b255b,DISK] 2023-02-11 01:00:53,606 INFO [RS:2;jenkins-hbase4:42073] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/593564ae-939c-6914-503a-94843f3f7058/WALs/jenkins-hbase4.apache.org,42073,1676077252069/jenkins-hbase4.apache.org%2C42073%2C1676077252069.1676077253545 2023-02-11 01:00:53,606 INFO [RS:1;jenkins-hbase4:44999] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/593564ae-939c-6914-503a-94843f3f7058/WALs/jenkins-hbase4.apache.org,44999,1676077252037/jenkins-hbase4.apache.org%2C44999%2C1676077252037.1676077253545 2023-02-11 01:00:53,607 DEBUG [RS:2;jenkins-hbase4:42073] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:39915,DS-9ebc61aa-9b9c-41d9-abc9-b4b9b6de6337,DISK], DatanodeInfoWithStorage[127.0.0.1:37515,DS-19365098-07a0-4163-8732-27153e4b255b,DISK], DatanodeInfoWithStorage[127.0.0.1:39157,DS-8f401005-340e-4621-a18e-f461eb4d4ab2,DISK]] 2023-02-11 01:00:53,607 DEBUG [RS:1;jenkins-hbase4:44999] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:37515,DS-19365098-07a0-4163-8732-27153e4b255b,DISK], DatanodeInfoWithStorage[127.0.0.1:39915,DS-9ebc61aa-9b9c-41d9-abc9-b4b9b6de6337,DISK], DatanodeInfoWithStorage[127.0.0.1:39157,DS-8f401005-340e-4621-a18e-f461eb4d4ab2,DISK]] 2023-02-11 01:00:53,630 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,42073,1676077252069 2023-02-11 01:00:53,633 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-02-11 01:00:53,635 INFO [RS-EventLoopGroup-5-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:54678, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-02-11 01:00:53,648 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open hbase:meta,,1.1588230740 2023-02-11 01:00:53,648 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-11 01:00:53,652 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C42073%2C1676077252069.meta, suffix=.meta, logDir=hdfs://localhost:44295/user/jenkins/test-data/593564ae-939c-6914-503a-94843f3f7058/WALs/jenkins-hbase4.apache.org,42073,1676077252069, archiveDir=hdfs://localhost:44295/user/jenkins/test-data/593564ae-939c-6914-503a-94843f3f7058/oldWALs, maxLogs=32 2023-02-11 01:00:53,672 DEBUG [RS-EventLoopGroup-5-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:39915,DS-9ebc61aa-9b9c-41d9-abc9-b4b9b6de6337,DISK] 2023-02-11 01:00:53,672 DEBUG [RS-EventLoopGroup-5-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:39157,DS-8f401005-340e-4621-a18e-f461eb4d4ab2,DISK] 2023-02-11 01:00:53,673 DEBUG [RS-EventLoopGroup-5-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:37515,DS-19365098-07a0-4163-8732-27153e4b255b,DISK] 2023-02-11 01:00:53,678 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/593564ae-939c-6914-503a-94843f3f7058/WALs/jenkins-hbase4.apache.org,42073,1676077252069/jenkins-hbase4.apache.org%2C42073%2C1676077252069.meta.1676077253653.meta 2023-02-11 01:00:53,678 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:39915,DS-9ebc61aa-9b9c-41d9-abc9-b4b9b6de6337,DISK], DatanodeInfoWithStorage[127.0.0.1:39157,DS-8f401005-340e-4621-a18e-f461eb4d4ab2,DISK], DatanodeInfoWithStorage[127.0.0.1:37515,DS-19365098-07a0-4163-8732-27153e4b255b,DISK]] 2023-02-11 01:00:53,678 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-02-11 01:00:53,680 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-02-11 01:00:53,697 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(8546): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-02-11 01:00:53,702 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-02-11 01:00:53,708 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-02-11 01:00:53,709 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 01:00:53,709 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 1588230740 2023-02-11 01:00:53,709 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 1588230740 2023-02-11 01:00:53,712 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-02-11 01:00:53,713 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44295/user/jenkins/test-data/593564ae-939c-6914-503a-94843f3f7058/data/hbase/meta/1588230740/info 2023-02-11 01:00:53,713 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44295/user/jenkins/test-data/593564ae-939c-6914-503a-94843f3f7058/data/hbase/meta/1588230740/info 2023-02-11 01:00:53,714 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-02-11 01:00:53,715 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 01:00:53,715 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-02-11 01:00:53,716 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44295/user/jenkins/test-data/593564ae-939c-6914-503a-94843f3f7058/data/hbase/meta/1588230740/rep_barrier 2023-02-11 01:00:53,717 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44295/user/jenkins/test-data/593564ae-939c-6914-503a-94843f3f7058/data/hbase/meta/1588230740/rep_barrier 2023-02-11 01:00:53,717 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-02-11 01:00:53,718 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 01:00:53,718 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-02-11 01:00:53,719 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44295/user/jenkins/test-data/593564ae-939c-6914-503a-94843f3f7058/data/hbase/meta/1588230740/table 2023-02-11 01:00:53,719 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44295/user/jenkins/test-data/593564ae-939c-6914-503a-94843f3f7058/data/hbase/meta/1588230740/table 2023-02-11 01:00:53,720 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-02-11 01:00:53,721 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 01:00:53,724 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:44295/user/jenkins/test-data/593564ae-939c-6914-503a-94843f3f7058/data/hbase/meta/1588230740 2023-02-11 01:00:53,729 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:44295/user/jenkins/test-data/593564ae-939c-6914-503a-94843f3f7058/data/hbase/meta/1588230740 2023-02-11 01:00:53,732 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-02-11 01:00:53,734 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 1588230740 2023-02-11 01:00:53,736 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 1588230740; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=72744283, jitterRate=0.08397428691387177}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-02-11 01:00:53,736 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 1588230740: Running coprocessor pre-open hook at 1676077253709Writing region info on filesystem at 1676077253709Initializing all the Stores at 1676077253711 (+2 ms)Instantiating store for column family {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} at 1676077253711Instantiating store for column family {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676077253711Instantiating store for column family {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} at 1676077253711Cleaning up temporary data from old regions at 1676077253730 (+19 ms)Cleaning up detritus from prior splits at 1676077253731 (+1 ms)Running coprocessor post-open hooks at 1676077253736 (+5 ms)Region opened successfully at 1676077253736 2023-02-11 01:00:53,746 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:meta,,1.1588230740, pid=3, masterSystemTime=1676077253623 2023-02-11 01:00:53,763 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:meta,,1.1588230740 2023-02-11 01:00:53,764 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened hbase:meta,,1.1588230740 2023-02-11 01:00:53,764 INFO [PEWorker-5] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,42073,1676077252069, state=OPEN 2023-02-11 01:00:53,767 DEBUG [Listener at localhost/46601-EventThread] zookeeper.ZKWatcher(600): master:33145-0x10040e51f9c0000, quorum=127.0.0.1:63347, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-02-11 01:00:53,767 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-02-11 01:00:53,774 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=3, resume processing ppid=2 2023-02-11 01:00:53,774 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=3, ppid=2, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,42073,1676077252069 in 334 msec 2023-02-11 01:00:53,779 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=2, resume processing ppid=1 2023-02-11 01:00:53,780 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=2, ppid=1, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 539 msec 2023-02-11 01:00:53,785 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta in 777 msec 2023-02-11 01:00:53,785 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(946): Master startup: status=Wait for region servers to report in, state=RUNNING, startTime=1676077252111, completionTime=-1 2023-02-11 01:00:53,785 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ServerManager(821): Finished waiting on RegionServer count=3; waited=0ms, expected min=3 server(s), max=3 server(s), master is running 2023-02-11 01:00:53,785 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1519): Joining cluster... 2023-02-11 01:00:53,851 DEBUG [hconnection-0x3e24d6e7-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-11 01:00:53,854 INFO [RS-EventLoopGroup-5-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:54682, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-11 01:00:53,870 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1531): Number of RegionServers=3 2023-02-11 01:00:53,870 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1676077313870 2023-02-11 01:00:53,870 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1676077373870 2023-02-11 01:00:53,870 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1538): Joined the cluster in 84 msec 2023-02-11 01:00:53,893 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,33145,1676077250844-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2023-02-11 01:00:53,893 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,33145,1676077250844-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2023-02-11 01:00:53,893 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,33145,1676077250844-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2023-02-11 01:00:53,895 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=CatalogJanitor-jenkins-hbase4:33145, period=300000, unit=MILLISECONDS is enabled. 2023-02-11 01:00:53,895 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2023-02-11 01:00:53,905 DEBUG [master/jenkins-hbase4:0.Chore.1] janitor.CatalogJanitor(175): 2023-02-11 01:00:53,915 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.TableNamespaceManager(92): Namespace table not found. Creating... 2023-02-11 01:00:53,916 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2138): Client=null/null create 'hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-02-11 01:00:53,927 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2023-02-11 01:00:53,931 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_PRE_OPERATION 2023-02-11 01:00:53,934 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-02-11 01:00:53,955 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:44295/user/jenkins/test-data/593564ae-939c-6914-503a-94843f3f7058/.tmp/data/hbase/namespace/568ad5dacf91067899b20069681f9127 2023-02-11 01:00:53,958 DEBUG [HFileArchiver-1] backup.HFileArchiver(153): Directory hdfs://localhost:44295/user/jenkins/test-data/593564ae-939c-6914-503a-94843f3f7058/.tmp/data/hbase/namespace/568ad5dacf91067899b20069681f9127 empty. 2023-02-11 01:00:53,958 DEBUG [HFileArchiver-1] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:44295/user/jenkins/test-data/593564ae-939c-6914-503a-94843f3f7058/.tmp/data/hbase/namespace/568ad5dacf91067899b20069681f9127 2023-02-11 01:00:53,958 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived hbase:namespace regions 2023-02-11 01:00:53,993 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:44295/user/jenkins/test-data/593564ae-939c-6914-503a-94843f3f7058/.tmp/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 2023-02-11 01:00:53,996 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(7671): creating {ENCODED => 568ad5dacf91067899b20069681f9127, NAME => 'hbase:namespace,,1676077253916.568ad5dacf91067899b20069681f9127.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:44295/user/jenkins/test-data/593564ae-939c-6914-503a-94843f3f7058/.tmp 2023-02-11 01:00:54,014 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(865): Instantiated hbase:namespace,,1676077253916.568ad5dacf91067899b20069681f9127.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 01:00:54,015 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1603): Closing 568ad5dacf91067899b20069681f9127, disabling compactions & flushes 2023-02-11 01:00:54,015 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1625): Closing region hbase:namespace,,1676077253916.568ad5dacf91067899b20069681f9127. 2023-02-11 01:00:54,015 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1646): Waiting without time limit for close lock on hbase:namespace,,1676077253916.568ad5dacf91067899b20069681f9127. 2023-02-11 01:00:54,015 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1713): Acquired close lock on hbase:namespace,,1676077253916.568ad5dacf91067899b20069681f9127. after waiting 0 ms 2023-02-11 01:00:54,015 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1723): Updates disabled for region hbase:namespace,,1676077253916.568ad5dacf91067899b20069681f9127. 2023-02-11 01:00:54,015 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1837): Closed hbase:namespace,,1676077253916.568ad5dacf91067899b20069681f9127. 2023-02-11 01:00:54,016 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1557): Region close journal for 568ad5dacf91067899b20069681f9127: Waiting for close lock at 1676077254015Disabling compacts and flushes for region at 1676077254015Disabling writes for close at 1676077254015Writing region close event to WAL at 1676077254015Closed at 1676077254015 2023-02-11 01:00:54,021 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ADD_TO_META 2023-02-11 01:00:54,037 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"hbase:namespace,,1676077253916.568ad5dacf91067899b20069681f9127.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1676077254023"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676077254023"}]},"ts":"1676077254023"} 2023-02-11 01:00:54,065 INFO [PEWorker-3] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-02-11 01:00:54,066 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-02-11 01:00:54,071 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676077254066"}]},"ts":"1676077254066"} 2023-02-11 01:00:54,075 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLING in hbase:meta 2023-02-11 01:00:54,079 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-02-11 01:00:54,080 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-11 01:00:54,080 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-11 01:00:54,080 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-11 01:00:54,080 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-11 01:00:54,083 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=568ad5dacf91067899b20069681f9127, ASSIGN}] 2023-02-11 01:00:54,085 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=568ad5dacf91067899b20069681f9127, ASSIGN 2023-02-11 01:00:54,087 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=568ad5dacf91067899b20069681f9127, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,42073,1676077252069; forceNewPlan=false, retain=false 2023-02-11 01:00:54,239 INFO [jenkins-hbase4:33145] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-02-11 01:00:54,240 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=568ad5dacf91067899b20069681f9127, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,42073,1676077252069 2023-02-11 01:00:54,240 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1676077253916.568ad5dacf91067899b20069681f9127.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1676077254240"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676077254240"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676077254240"}]},"ts":"1676077254240"} 2023-02-11 01:00:54,244 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=6, ppid=5, state=RUNNABLE; OpenRegionProcedure 568ad5dacf91067899b20069681f9127, server=jenkins-hbase4.apache.org,42073,1676077252069}] 2023-02-11 01:00:54,406 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open hbase:namespace,,1676077253916.568ad5dacf91067899b20069681f9127. 2023-02-11 01:00:54,407 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 568ad5dacf91067899b20069681f9127, NAME => 'hbase:namespace,,1676077253916.568ad5dacf91067899b20069681f9127.', STARTKEY => '', ENDKEY => ''} 2023-02-11 01:00:54,408 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace 568ad5dacf91067899b20069681f9127 2023-02-11 01:00:54,408 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated hbase:namespace,,1676077253916.568ad5dacf91067899b20069681f9127.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 01:00:54,408 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 568ad5dacf91067899b20069681f9127 2023-02-11 01:00:54,408 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 568ad5dacf91067899b20069681f9127 2023-02-11 01:00:54,410 INFO [StoreOpener-568ad5dacf91067899b20069681f9127-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 568ad5dacf91067899b20069681f9127 2023-02-11 01:00:54,412 DEBUG [StoreOpener-568ad5dacf91067899b20069681f9127-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44295/user/jenkins/test-data/593564ae-939c-6914-503a-94843f3f7058/data/hbase/namespace/568ad5dacf91067899b20069681f9127/info 2023-02-11 01:00:54,412 DEBUG [StoreOpener-568ad5dacf91067899b20069681f9127-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:44295/user/jenkins/test-data/593564ae-939c-6914-503a-94843f3f7058/data/hbase/namespace/568ad5dacf91067899b20069681f9127/info 2023-02-11 01:00:54,413 INFO [StoreOpener-568ad5dacf91067899b20069681f9127-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 568ad5dacf91067899b20069681f9127 columnFamilyName info 2023-02-11 01:00:54,414 INFO [StoreOpener-568ad5dacf91067899b20069681f9127-1] regionserver.HStore(310): Store=568ad5dacf91067899b20069681f9127/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 01:00:54,415 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:44295/user/jenkins/test-data/593564ae-939c-6914-503a-94843f3f7058/data/hbase/namespace/568ad5dacf91067899b20069681f9127 2023-02-11 01:00:54,416 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:44295/user/jenkins/test-data/593564ae-939c-6914-503a-94843f3f7058/data/hbase/namespace/568ad5dacf91067899b20069681f9127 2023-02-11 01:00:54,421 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 568ad5dacf91067899b20069681f9127 2023-02-11 01:00:54,424 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44295/user/jenkins/test-data/593564ae-939c-6914-503a-94843f3f7058/data/hbase/namespace/568ad5dacf91067899b20069681f9127/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-11 01:00:54,425 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 568ad5dacf91067899b20069681f9127; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=72702415, jitterRate=0.08335040509700775}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 01:00:54,425 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 568ad5dacf91067899b20069681f9127: Running coprocessor pre-open hook at 1676077254408Writing region info on filesystem at 1676077254408Initializing all the Stores at 1676077254409 (+1 ms)Instantiating store for column family {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} at 1676077254409Cleaning up temporary data from old regions at 1676077254417 (+8 ms)Cleaning up detritus from prior splits at 1676077254418 (+1 ms)Running coprocessor post-open hooks at 1676077254425 (+7 ms)Region opened successfully at 1676077254425 2023-02-11 01:00:54,427 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:namespace,,1676077253916.568ad5dacf91067899b20069681f9127., pid=6, masterSystemTime=1676077254398 2023-02-11 01:00:54,430 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:namespace,,1676077253916.568ad5dacf91067899b20069681f9127. 2023-02-11 01:00:54,430 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened hbase:namespace,,1676077253916.568ad5dacf91067899b20069681f9127. 2023-02-11 01:00:54,432 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=568ad5dacf91067899b20069681f9127, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,42073,1676077252069 2023-02-11 01:00:54,432 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1676077253916.568ad5dacf91067899b20069681f9127.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1676077254431"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676077254431"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676077254431"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676077254431"}]},"ts":"1676077254431"} 2023-02-11 01:00:54,439 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=6, resume processing ppid=5 2023-02-11 01:00:54,439 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=6, ppid=5, state=SUCCESS; OpenRegionProcedure 568ad5dacf91067899b20069681f9127, server=jenkins-hbase4.apache.org,42073,1676077252069 in 191 msec 2023-02-11 01:00:54,443 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=5, resume processing ppid=4 2023-02-11 01:00:54,443 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=5, ppid=4, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=568ad5dacf91067899b20069681f9127, ASSIGN in 357 msec 2023-02-11 01:00:54,445 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-02-11 01:00:54,445 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676077254445"}]},"ts":"1676077254445"} 2023-02-11 01:00:54,447 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLED in hbase:meta 2023-02-11 01:00:54,451 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_POST_OPERATION 2023-02-11 01:00:54,454 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace in 534 msec 2023-02-11 01:00:54,531 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:33145-0x10040e51f9c0000, quorum=127.0.0.1:63347, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/namespace 2023-02-11 01:00:54,533 DEBUG [Listener at localhost/46601-EventThread] zookeeper.ZKWatcher(600): master:33145-0x10040e51f9c0000, quorum=127.0.0.1:63347, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/namespace 2023-02-11 01:00:54,533 DEBUG [Listener at localhost/46601-EventThread] zookeeper.ZKWatcher(600): master:33145-0x10040e51f9c0000, quorum=127.0.0.1:63347, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-11 01:00:54,566 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=7, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2023-02-11 01:00:54,580 DEBUG [Listener at localhost/46601-EventThread] zookeeper.ZKWatcher(600): master:33145-0x10040e51f9c0000, quorum=127.0.0.1:63347, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-02-11 01:00:54,586 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default in 27 msec 2023-02-11 01:00:54,589 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=8, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2023-02-11 01:00:54,601 DEBUG [Listener at localhost/46601-EventThread] zookeeper.ZKWatcher(600): master:33145-0x10040e51f9c0000, quorum=127.0.0.1:63347, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-02-11 01:00:54,606 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase in 16 msec 2023-02-11 01:00:54,614 DEBUG [Listener at localhost/46601-EventThread] zookeeper.ZKWatcher(600): master:33145-0x10040e51f9c0000, quorum=127.0.0.1:63347, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-02-11 01:00:54,618 DEBUG [Listener at localhost/46601-EventThread] zookeeper.ZKWatcher(600): master:33145-0x10040e51f9c0000, quorum=127.0.0.1:63347, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2023-02-11 01:00:54,619 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1077): Master has completed initialization 2.496sec 2023-02-11 01:00:54,622 INFO [master/jenkins-hbase4:0:becomeActiveMaster] quotas.MasterQuotaManager(97): Quota support disabled 2023-02-11 01:00:54,624 INFO [master/jenkins-hbase4:0:becomeActiveMaster] slowlog.SlowLogMasterService(57): Slow/Large requests logging to system table hbase:slowlog is disabled. Quitting. 2023-02-11 01:00:54,624 INFO [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKWatcher(269): not a secure deployment, proceeding 2023-02-11 01:00:54,625 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,33145,1676077250844-ExpiredMobFileCleanerChore, period=86400, unit=SECONDS is enabled. 2023-02-11 01:00:54,627 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,33145,1676077250844-MobCompactionChore, period=604800, unit=SECONDS is enabled. 2023-02-11 01:00:54,635 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1166): Balancer post startup initialization complete, took 0 seconds 2023-02-11 01:00:54,699 DEBUG [Listener at localhost/46601] zookeeper.ReadOnlyZKClient(139): Connect 0x0491b409 to 127.0.0.1:63347 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-11 01:00:54,704 DEBUG [Listener at localhost/46601] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4ff25707, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-11 01:00:54,718 DEBUG [hconnection-0x4ca3451-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-11 01:00:54,731 INFO [RS-EventLoopGroup-5-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:54686, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-11 01:00:54,741 INFO [Listener at localhost/46601] hbase.HBaseTestingUtility(1145): Minicluster is up; activeMaster=jenkins-hbase4.apache.org,33145,1676077250844 2023-02-11 01:00:54,741 DEBUG [Listener at localhost/46601] zookeeper.ReadOnlyZKClient(139): Connect 0x35e88b89 to 127.0.0.1:63347 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-11 01:00:54,754 DEBUG [ReadOnlyZKClient-127.0.0.1:63347@0x35e88b89] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6abd6126, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-11 01:00:54,821 DEBUG [Listener at localhost/46601] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-02-11 01:00:54,824 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:49108, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-02-11 01:00:54,825 INFO [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=41571] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,41571,1676077251993' ***** 2023-02-11 01:00:54,826 INFO [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=41571] regionserver.HRegionServer(2310): STOPPED: Called by admin client org.apache.hadoop.hbase.client.AsyncConnectionImpl@2cd71b0a 2023-02-11 01:00:54,826 INFO [RS:0;jenkins-hbase4:41571] regionserver.HeapMemoryManager(220): Stopping 2023-02-11 01:00:54,827 INFO [RS:0;jenkins-hbase4:41571] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-02-11 01:00:54,827 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-02-11 01:00:54,827 INFO [RS:0;jenkins-hbase4:41571] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-02-11 01:00:54,828 INFO [RS:0;jenkins-hbase4:41571] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,41571,1676077251993 2023-02-11 01:00:54,828 DEBUG [RS:0;jenkins-hbase4:41571] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0d01c485 to 127.0.0.1:63347 2023-02-11 01:00:54,828 DEBUG [RS:0;jenkins-hbase4:41571] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-11 01:00:54,828 INFO [RS:0;jenkins-hbase4:41571] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,41571,1676077251993; all regions closed. 2023-02-11 01:00:54,830 DEBUG [Listener at localhost/46601] client.ConnectionUtils(586): Start fetching master stub from registry 2023-02-11 01:00:54,834 DEBUG [ReadOnlyZKClient-127.0.0.1:63347@0x35e88b89] client.AsyncConnectionImpl(289): The fetched master address is jenkins-hbase4.apache.org,33145,1676077250844 2023-02-11 01:00:54,838 DEBUG [ReadOnlyZKClient-127.0.0.1:63347@0x35e88b89] client.ConnectionUtils(594): The fetched master stub is org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$Stub@73275879 2023-02-11 01:00:54,843 DEBUG [ReadOnlyZKClient-127.0.0.1:63347@0x35e88b89] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-02-11 01:00:54,847 DEBUG [RS:0;jenkins-hbase4:41571] wal.AbstractFSWAL(932): Moved 1 WAL file(s) to /user/jenkins/test-data/593564ae-939c-6914-503a-94843f3f7058/oldWALs 2023-02-11 01:00:54,848 INFO [RS:0;jenkins-hbase4:41571] wal.AbstractFSWAL(935): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C41571%2C1676077251993:(num 1676077253545) 2023-02-11 01:00:54,848 DEBUG [RS:0;jenkins-hbase4:41571] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-11 01:00:54,848 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:37738, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-02-11 01:00:54,848 INFO [RS:0;jenkins-hbase4:41571] regionserver.LeaseManager(133): Closed leases 2023-02-11 01:00:54,848 INFO [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=33145] master.MasterRpcServices(1601): Client=jenkins//172.31.14.131 stop 2023-02-11 01:00:54,850 INFO [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=33145] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,33145,1676077250844' ***** 2023-02-11 01:00:54,850 INFO [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=33145] regionserver.HRegionServer(2310): STOPPED: Stopped by RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=33145 2023-02-11 01:00:54,850 INFO [RS:0;jenkins-hbase4:41571] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS, ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS] on shutdown 2023-02-11 01:00:54,850 DEBUG [M:0;jenkins-hbase4:33145] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2b7e7350, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-02-11 01:00:54,850 INFO [RS:0;jenkins-hbase4:41571] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-02-11 01:00:54,850 INFO [M:0;jenkins-hbase4:33145] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,33145,1676077250844 2023-02-11 01:00:54,850 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-02-11 01:00:54,850 INFO [RS:0;jenkins-hbase4:41571] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-02-11 01:00:54,851 INFO [RS:0;jenkins-hbase4:41571] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-02-11 01:00:54,851 INFO [RS:0;jenkins-hbase4:41571] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:41571 2023-02-11 01:00:54,852 DEBUG [M:0;jenkins-hbase4:33145] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2afa76c5 to 127.0.0.1:63347 2023-02-11 01:00:54,852 DEBUG [M:0;jenkins-hbase4:33145] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-11 01:00:54,852 INFO [M:0;jenkins-hbase4:33145] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,33145,1676077250844; all regions closed. 2023-02-11 01:00:54,852 DEBUG [M:0;jenkins-hbase4:33145] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-11 01:00:54,852 DEBUG [M:0;jenkins-hbase4:33145] cleaner.LogCleaner(198): Cancelling LogCleaner 2023-02-11 01:00:54,852 WARN [OldWALsCleaner-0] cleaner.LogCleaner(186): Interrupted while cleaning old WALs, will try to clean it next round. Exiting. 2023-02-11 01:00:54,852 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1676077253144] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1676077253144,5,FailOnTimeoutGroup] 2023-02-11 01:00:54,852 DEBUG [M:0;jenkins-hbase4:33145] cleaner.HFileCleaner(317): Stopping file delete threads 2023-02-11 01:00:54,852 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1676077253144] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1676077253144,5,FailOnTimeoutGroup] 2023-02-11 01:00:54,853 INFO [M:0;jenkins-hbase4:33145] master.MasterMobCompactionThread(168): Waiting for Mob Compaction Thread to finish... 2023-02-11 01:00:54,853 INFO [M:0;jenkins-hbase4:33145] master.MasterMobCompactionThread(168): Waiting for Region Server Mob Compaction Thread to finish... 2023-02-11 01:00:54,854 DEBUG [Listener at localhost/46601-EventThread] zookeeper.ZKWatcher(600): regionserver:41571-0x10040e51f9c0001, quorum=127.0.0.1:63347, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-02-11 01:00:54,854 DEBUG [Listener at localhost/46601-EventThread] zookeeper.ZKWatcher(600): regionserver:44999-0x10040e51f9c0002, quorum=127.0.0.1:63347, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-02-11 01:00:54,854 DEBUG [Listener at localhost/46601-EventThread] zookeeper.ZKWatcher(600): regionserver:42073-0x10040e51f9c0003, quorum=127.0.0.1:63347, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-02-11 01:00:54,854 INFO [M:0;jenkins-hbase4:33145] hbase.ChoreService(369): Chore service for: master/jenkins-hbase4:0 had [] on shutdown 2023-02-11 01:00:54,854 DEBUG [Listener at localhost/46601-EventThread] zookeeper.ZKWatcher(600): master:33145-0x10040e51f9c0000, quorum=127.0.0.1:63347, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-02-11 01:00:54,855 DEBUG [M:0;jenkins-hbase4:33145] master.HMaster(1502): Stopping service threads 2023-02-11 01:00:54,855 DEBUG [Listener at localhost/46601-EventThread] zookeeper.ZKWatcher(600): master:33145-0x10040e51f9c0000, quorum=127.0.0.1:63347, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-11 01:00:54,855 INFO [M:0;jenkins-hbase4:33145] procedure2.RemoteProcedureDispatcher(118): Stopping procedure remote dispatcher 2023-02-11 01:00:54,856 INFO [M:0;jenkins-hbase4:33145] procedure2.ProcedureExecutor(629): Stopping 2023-02-11 01:00:54,857 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42073-0x10040e51f9c0003, quorum=127.0.0.1:63347, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-11 01:00:54,857 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:33145-0x10040e51f9c0000, quorum=127.0.0.1:63347, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-11 01:00:54,857 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:41571-0x10040e51f9c0001, quorum=127.0.0.1:63347, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-11 01:00:54,857 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:44999-0x10040e51f9c0002, quorum=127.0.0.1:63347, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-11 01:00:54,857 ERROR [M:0;jenkins-hbase4:33145] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] Thread[HFileArchiver-1,5,PEWorkerGroup] 2023-02-11 01:00:54,857 INFO [M:0;jenkins-hbase4:33145] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-02-11 01:00:54,858 DEBUG [normalizer-worker-0] normalizer.RegionNormalizerWorker(174): interrupt detected. terminating. 2023-02-11 01:00:54,858 INFO [Listener at localhost/46601] client.AsyncConnectionImpl(207): Connection has been closed by Listener at localhost/46601. 2023-02-11 01:00:54,858 DEBUG [M:0;jenkins-hbase4:33145] zookeeper.ZKUtil(398): master:33145-0x10040e51f9c0000, quorum=127.0.0.1:63347, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2023-02-11 01:00:54,858 WARN [M:0;jenkins-hbase4:33145] master.ActiveMasterManager(323): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2023-02-11 01:00:54,858 DEBUG [Listener at localhost/46601] client.AsyncConnectionImpl(232): Call stack: at java.lang.Thread.getStackTrace(Thread.java:1564) at org.apache.hadoop.hbase.client.AsyncConnectionImpl.close(AsyncConnectionImpl.java:209) at org.apache.hbase.thirdparty.com.google.common.io.Closeables.close(Closeables.java:79) at org.apache.hadoop.hbase.client.TestAsyncClusterAdminApi2.tearDown(TestAsyncClusterAdminApi2.java:75) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56) at org.junit.internal.runners.statements.RunAfters.invokeMethod(RunAfters.java:46) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:33) at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61) at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306) at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63) at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329) at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.apache.hadoop.hbase.SystemExitRule$1.evaluate(SystemExitRule.java:39) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:750) 2023-02-11 01:00:54,858 INFO [M:0;jenkins-hbase4:33145] assignment.AssignmentManager(315): Stopping assignment manager 2023-02-11 01:00:54,859 INFO [M:0;jenkins-hbase4:33145] region.MasterRegion(167): Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=false 2023-02-11 01:00:54,859 DEBUG [M:0;jenkins-hbase4:33145] regionserver.HRegion(1603): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-02-11 01:00:54,859 INFO [M:0;jenkins-hbase4:33145] regionserver.HRegion(1625): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-11 01:00:54,859 DEBUG [M:0;jenkins-hbase4:33145] regionserver.HRegion(1646): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-11 01:00:54,859 DEBUG [M:0;jenkins-hbase4:33145] regionserver.HRegion(1713): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-02-11 01:00:54,859 DEBUG [M:0;jenkins-hbase4:33145] regionserver.HRegion(1723): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-11 01:00:54,861 INFO [M:0;jenkins-hbase4:33145] regionserver.HRegion(2744): Flushing 1595e783b53d99cd5eef43b6debb2682 1/1 column families, dataSize=24.07 KB heapSize=29.55 KB 2023-02-11 01:00:54,866 DEBUG [Listener at localhost/46601-EventThread] zookeeper.ZKWatcher(600): regionserver:42073-0x10040e51f9c0003, quorum=127.0.0.1:63347, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,41571,1676077251993 2023-02-11 01:00:54,866 DEBUG [Listener at localhost/46601-EventThread] zookeeper.ZKWatcher(600): master:33145-0x10040e51f9c0000, quorum=127.0.0.1:63347, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-11 01:00:54,866 DEBUG [Listener at localhost/46601-EventThread] zookeeper.ZKWatcher(600): regionserver:42073-0x10040e51f9c0003, quorum=127.0.0.1:63347, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-11 01:00:54,866 DEBUG [Listener at localhost/46601-EventThread] zookeeper.ZKWatcher(600): regionserver:41571-0x10040e51f9c0001, quorum=127.0.0.1:63347, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,41571,1676077251993 2023-02-11 01:00:54,866 DEBUG [Listener at localhost/46601-EventThread] zookeeper.ZKWatcher(600): regionserver:44999-0x10040e51f9c0002, quorum=127.0.0.1:63347, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,41571,1676077251993 2023-02-11 01:00:54,866 DEBUG [Listener at localhost/46601-EventThread] zookeeper.ZKWatcher(600): regionserver:44999-0x10040e51f9c0002, quorum=127.0.0.1:63347, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-11 01:00:54,866 ERROR [Listener at localhost/46601-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@4436ffb6 rejected from java.util.concurrent.ThreadPoolExecutor@400973e8[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-11 01:00:54,867 DEBUG [Listener at localhost/46601-EventThread] zookeeper.ZKWatcher(600): regionserver:41571-0x10040e51f9c0001, quorum=127.0.0.1:63347, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-11 01:00:54,867 ERROR [Listener at localhost/46601-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@477c186c rejected from java.util.concurrent.ThreadPoolExecutor@400973e8[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-11 01:00:54,869 DEBUG [Listener at localhost/46601] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-11 01:00:54,869 DEBUG [Listener at localhost/46601] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x35e88b89 to 127.0.0.1:63347 2023-02-11 01:00:54,869 INFO [Listener at localhost/46601] hbase.HBaseTestingUtility(1286): Shutting down minicluster 2023-02-11 01:00:54,870 DEBUG [Listener at localhost/46601] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0491b409 to 127.0.0.1:63347 2023-02-11 01:00:54,870 DEBUG [Listener at localhost/46601] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-11 01:00:54,870 DEBUG [Listener at localhost/46601] util.JVMClusterUtil(237): Shutting down HBase Cluster 2023-02-11 01:00:54,871 INFO [Listener at localhost/46601] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,44999,1676077252037' ***** 2023-02-11 01:00:54,871 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:44999-0x10040e51f9c0002, quorum=127.0.0.1:63347, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,42073,1676077252069 2023-02-11 01:00:54,871 INFO [Listener at localhost/46601] regionserver.HRegionServer(2310): STOPPED: Shutdown requested 2023-02-11 01:00:54,871 INFO [Listener at localhost/46601] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,42073,1676077252069' ***** 2023-02-11 01:00:54,871 INFO [Listener at localhost/46601] regionserver.HRegionServer(2310): STOPPED: Shutdown requested 2023-02-11 01:00:54,871 INFO [RS:1;jenkins-hbase4:44999] regionserver.HeapMemoryManager(220): Stopping 2023-02-11 01:00:54,871 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-02-11 01:00:54,871 INFO [RS:1;jenkins-hbase4:44999] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-02-11 01:00:54,872 INFO [RS:1;jenkins-hbase4:44999] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-02-11 01:00:54,872 INFO [RS:2;jenkins-hbase4:42073] regionserver.HeapMemoryManager(220): Stopping 2023-02-11 01:00:54,872 INFO [RS:1;jenkins-hbase4:44999] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,44999,1676077252037 2023-02-11 01:00:54,872 INFO [RS:2;jenkins-hbase4:42073] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-02-11 01:00:54,872 DEBUG [RS:1;jenkins-hbase4:44999] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2a3b18e1 to 127.0.0.1:63347 2023-02-11 01:00:54,872 DEBUG [RS:1;jenkins-hbase4:44999] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-11 01:00:54,873 INFO [RS:1;jenkins-hbase4:44999] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,44999,1676077252037; all regions closed. 2023-02-11 01:00:54,873 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:44999-0x10040e51f9c0002, quorum=127.0.0.1:63347, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,44999,1676077252037 2023-02-11 01:00:54,872 INFO [RS:2;jenkins-hbase4:42073] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-02-11 01:00:54,872 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-02-11 01:00:54,874 INFO [zk-event-processor-pool-0] replication.ReplicationTrackerZKImpl$OtherRegionServerWatcher(124): /hbase/rs/jenkins-hbase4.apache.org,41571,1676077251993 znode expired, triggering replicatorRemoved event 2023-02-11 01:00:54,873 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:42073-0x10040e51f9c0003, quorum=127.0.0.1:63347, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,42073,1676077252069 2023-02-11 01:00:54,874 INFO [RS:2;jenkins-hbase4:42073] regionserver.HRegionServer(3304): Received CLOSE for 568ad5dacf91067899b20069681f9127 2023-02-11 01:00:54,874 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:42073-0x10040e51f9c0003, quorum=127.0.0.1:63347, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,44999,1676077252037 2023-02-11 01:00:54,877 INFO [zk-event-processor-pool-0] replication.ReplicationTrackerZKImpl$OtherRegionServerWatcher(124): /hbase/rs/jenkins-hbase4.apache.org,41571,1676077251993 znode expired, triggering replicatorRemoved event 2023-02-11 01:00:54,877 INFO [RS:2;jenkins-hbase4:42073] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,42073,1676077252069 2023-02-11 01:00:54,878 DEBUG [RS:2;jenkins-hbase4:42073] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x669fb05d to 127.0.0.1:63347 2023-02-11 01:00:54,878 DEBUG [RS:2;jenkins-hbase4:42073] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-11 01:00:54,878 INFO [RS:2;jenkins-hbase4:42073] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-02-11 01:00:54,878 INFO [RS:2;jenkins-hbase4:42073] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-02-11 01:00:54,878 INFO [RS:2;jenkins-hbase4:42073] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-02-11 01:00:54,878 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 568ad5dacf91067899b20069681f9127, disabling compactions & flushes 2023-02-11 01:00:54,878 INFO [RS:2;jenkins-hbase4:42073] regionserver.HRegionServer(3304): Received CLOSE for 1588230740 2023-02-11 01:00:54,879 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region hbase:namespace,,1676077253916.568ad5dacf91067899b20069681f9127. 2023-02-11 01:00:54,879 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on hbase:namespace,,1676077253916.568ad5dacf91067899b20069681f9127. 2023-02-11 01:00:54,879 INFO [RS:2;jenkins-hbase4:42073] regionserver.HRegionServer(1475): Waiting on 2 regions to close 2023-02-11 01:00:54,879 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on hbase:namespace,,1676077253916.568ad5dacf91067899b20069681f9127. after waiting 0 ms 2023-02-11 01:00:54,880 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 1588230740, disabling compactions & flushes 2023-02-11 01:00:54,880 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region hbase:namespace,,1676077253916.568ad5dacf91067899b20069681f9127. 2023-02-11 01:00:54,879 DEBUG [RS:2;jenkins-hbase4:42073] regionserver.HRegionServer(1479): Online Regions={568ad5dacf91067899b20069681f9127=hbase:namespace,,1676077253916.568ad5dacf91067899b20069681f9127., 1588230740=hbase:meta,,1.1588230740} 2023-02-11 01:00:54,880 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 568ad5dacf91067899b20069681f9127 1/1 column families, dataSize=78 B heapSize=488 B 2023-02-11 01:00:54,880 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region hbase:meta,,1.1588230740 2023-02-11 01:00:54,880 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-02-11 01:00:54,880 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-02-11 01:00:54,880 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region hbase:meta,,1.1588230740 2023-02-11 01:00:54,880 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 1588230740 3/3 column families, dataSize=1.26 KB heapSize=2.89 KB 2023-02-11 01:00:54,882 DEBUG [RS:2;jenkins-hbase4:42073] regionserver.HRegionServer(1505): Waiting on 1588230740, 568ad5dacf91067899b20069681f9127 2023-02-11 01:00:54,892 DEBUG [RS:1;jenkins-hbase4:44999] wal.AbstractFSWAL(932): Moved 1 WAL file(s) to /user/jenkins/test-data/593564ae-939c-6914-503a-94843f3f7058/oldWALs 2023-02-11 01:00:54,892 INFO [RS:1;jenkins-hbase4:44999] wal.AbstractFSWAL(935): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C44999%2C1676077252037:(num 1676077253545) 2023-02-11 01:00:54,892 DEBUG [RS:1;jenkins-hbase4:44999] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-11 01:00:54,892 INFO [RS:1;jenkins-hbase4:44999] regionserver.LeaseManager(133): Closed leases 2023-02-11 01:00:54,892 INFO [RS:1;jenkins-hbase4:44999] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS, ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS] on shutdown 2023-02-11 01:00:54,893 INFO [RS:1;jenkins-hbase4:44999] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-02-11 01:00:54,893 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-02-11 01:00:54,893 INFO [RS:1;jenkins-hbase4:44999] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-02-11 01:00:54,893 INFO [RS:1;jenkins-hbase4:44999] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-02-11 01:00:54,893 INFO [RS:1;jenkins-hbase4:44999] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:44999 2023-02-11 01:00:54,909 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-02-11 01:00:54,909 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-02-11 01:00:54,909 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-02-11 01:00:54,959 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=78 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost:44295/user/jenkins/test-data/593564ae-939c-6914-503a-94843f3f7058/data/hbase/namespace/568ad5dacf91067899b20069681f9127/.tmp/info/f9aa124f6e3844b88c9b8deaf83680c3 2023-02-11 01:00:54,960 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=1.17 KB at sequenceid=9 (bloomFilter=false), to=hdfs://localhost:44295/user/jenkins/test-data/593564ae-939c-6914-503a-94843f3f7058/data/hbase/meta/1588230740/.tmp/info/eb1d64753c804c89bf9b0a25b0e90524 2023-02-11 01:00:54,961 INFO [M:0;jenkins-hbase4:33145] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.07 KB at sequenceid=66 (bloomFilter=true), to=hdfs://localhost:44295/user/jenkins/test-data/593564ae-939c-6914-503a-94843f3f7058/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/8e79c10da2f04187b1434101f5ac1bd1 2023-02-11 01:00:54,978 DEBUG [Listener at localhost/46601-EventThread] zookeeper.ZKWatcher(600): regionserver:44999-0x10040e51f9c0002, quorum=127.0.0.1:63347, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,44999,1676077252037 2023-02-11 01:00:54,978 DEBUG [Listener at localhost/46601-EventThread] zookeeper.ZKWatcher(600): regionserver:42073-0x10040e51f9c0003, quorum=127.0.0.1:63347, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,44999,1676077252037 2023-02-11 01:00:54,978 ERROR [Listener at localhost/46601-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@4c4939ea rejected from java.util.concurrent.ThreadPoolExecutor@4876a4bb[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 6] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-11 01:00:54,978 DEBUG [Listener at localhost/46601-EventThread] zookeeper.ZKWatcher(600): regionserver:42073-0x10040e51f9c0003, quorum=127.0.0.1:63347, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-11 01:00:54,979 DEBUG [Listener at localhost/46601-EventThread] zookeeper.ZKWatcher(600): regionserver:44999-0x10040e51f9c0002, quorum=127.0.0.1:63347, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-11 01:00:54,979 ERROR [Listener at localhost/46601-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@39b04523 rejected from java.util.concurrent.ThreadPoolExecutor@4876a4bb[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 6] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-11 01:00:55,004 DEBUG [M:0;jenkins-hbase4:33145] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44295/user/jenkins/test-data/593564ae-939c-6914-503a-94843f3f7058/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/8e79c10da2f04187b1434101f5ac1bd1 as hdfs://localhost:44295/user/jenkins/test-data/593564ae-939c-6914-503a-94843f3f7058/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/8e79c10da2f04187b1434101f5ac1bd1 2023-02-11 01:00:55,004 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44295/user/jenkins/test-data/593564ae-939c-6914-503a-94843f3f7058/data/hbase/namespace/568ad5dacf91067899b20069681f9127/.tmp/info/f9aa124f6e3844b88c9b8deaf83680c3 as hdfs://localhost:44295/user/jenkins/test-data/593564ae-939c-6914-503a-94843f3f7058/data/hbase/namespace/568ad5dacf91067899b20069681f9127/info/f9aa124f6e3844b88c9b8deaf83680c3 2023-02-11 01:00:55,019 INFO [M:0;jenkins-hbase4:33145] regionserver.HStore(1080): Added hdfs://localhost:44295/user/jenkins/test-data/593564ae-939c-6914-503a-94843f3f7058/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/8e79c10da2f04187b1434101f5ac1bd1, entries=8, sequenceid=66, filesize=6.3 K 2023-02-11 01:00:55,021 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:44295/user/jenkins/test-data/593564ae-939c-6914-503a-94843f3f7058/data/hbase/namespace/568ad5dacf91067899b20069681f9127/info/f9aa124f6e3844b88c9b8deaf83680c3, entries=2, sequenceid=6, filesize=4.8 K 2023-02-11 01:00:55,024 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~78 B/78, heapSize ~472 B/472, currentSize=0 B/0 for 568ad5dacf91067899b20069681f9127 in 144ms, sequenceid=6, compaction requested=false 2023-02-11 01:00:55,024 INFO [M:0;jenkins-hbase4:33145] regionserver.HRegion(2947): Finished flush of dataSize ~24.07 KB/24646, heapSize ~29.54 KB/30248, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 163ms, sequenceid=66, compaction requested=false 2023-02-11 01:00:55,025 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:namespace' 2023-02-11 01:00:55,033 INFO [M:0;jenkins-hbase4:33145] regionserver.HRegion(1837): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-11 01:00:55,035 DEBUG [M:0;jenkins-hbase4:33145] regionserver.HRegion(1557): Region close journal for 1595e783b53d99cd5eef43b6debb2682: Waiting for close lock at 1676077254859Disabling compacts and flushes for region at 1676077254859Disabling writes for close at 1676077254859Obtaining lock to block concurrent updates at 1676077254861 (+2 ms)Preparing flush snapshotting stores in 1595e783b53d99cd5eef43b6debb2682 at 1676077254861Finished memstore snapshotting master:store,,1.1595e783b53d99cd5eef43b6debb2682., syncing WAL and waiting on mvcc, flushsize=dataSize=24646, getHeapSize=30248, getOffHeapSize=0, getCellsCount=71 at 1676077254877 (+16 ms)Flushing stores of master:store,,1.1595e783b53d99cd5eef43b6debb2682. at 1676077254880 (+3 ms)Flushing 1595e783b53d99cd5eef43b6debb2682/proc: creating writer at 1676077254885 (+5 ms)Flushing 1595e783b53d99cd5eef43b6debb2682/proc: appending metadata at 1676077254919 (+34 ms)Flushing 1595e783b53d99cd5eef43b6debb2682/proc: closing flushed file at 1676077254921 (+2 ms)Flushing 1595e783b53d99cd5eef43b6debb2682/proc: reopening flushed file at 1676077255006 (+85 ms)Finished flush of dataSize ~24.07 KB/24646, heapSize ~29.54 KB/30248, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 163ms, sequenceid=66, compaction requested=false at 1676077255024 (+18 ms)Writing region close event to WAL at 1676077255033 (+9 ms)Closed at 1676077255033 2023-02-11 01:00:55,043 INFO [M:0;jenkins-hbase4:33145] flush.MasterFlushTableProcedureManager(83): stop: server shutting down. 2023-02-11 01:00:55,044 INFO [M:0;jenkins-hbase4:33145] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:33145 2023-02-11 01:00:55,044 INFO [master:store-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-02-11 01:00:55,047 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=94 B at sequenceid=9 (bloomFilter=false), to=hdfs://localhost:44295/user/jenkins/test-data/593564ae-939c-6914-503a-94843f3f7058/data/hbase/meta/1588230740/.tmp/table/3dff8b881acc49558fa232e5fc413259 2023-02-11 01:00:55,047 DEBUG [M:0;jenkins-hbase4:33145] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase4.apache.org,33145,1676077250844 already deleted, retry=false 2023-02-11 01:00:55,049 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44295/user/jenkins/test-data/593564ae-939c-6914-503a-94843f3f7058/data/hbase/namespace/568ad5dacf91067899b20069681f9127/recovered.edits/9.seqid, newMaxSeqId=9, maxSeqId=1 2023-02-11 01:00:55,049 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed hbase:namespace,,1676077253916.568ad5dacf91067899b20069681f9127. 2023-02-11 01:00:55,050 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 568ad5dacf91067899b20069681f9127: Waiting for close lock at 1676077254878Running coprocessor pre-close hooks at 1676077254878Disabling compacts and flushes for region at 1676077254878Disabling writes for close at 1676077254880 (+2 ms)Obtaining lock to block concurrent updates at 1676077254880Preparing flush snapshotting stores in 568ad5dacf91067899b20069681f9127 at 1676077254880Finished memstore snapshotting hbase:namespace,,1676077253916.568ad5dacf91067899b20069681f9127., syncing WAL and waiting on mvcc, flushsize=dataSize=78, getHeapSize=472, getOffHeapSize=0, getCellsCount=2 at 1676077254881 (+1 ms)Flushing stores of hbase:namespace,,1676077253916.568ad5dacf91067899b20069681f9127. at 1676077254883 (+2 ms)Flushing 568ad5dacf91067899b20069681f9127/info: creating writer at 1676077254885 (+2 ms)Flushing 568ad5dacf91067899b20069681f9127/info: appending metadata at 1676077254918 (+33 ms)Flushing 568ad5dacf91067899b20069681f9127/info: closing flushed file at 1676077254921 (+3 ms)Flushing 568ad5dacf91067899b20069681f9127/info: reopening flushed file at 1676077255006 (+85 ms)Finished flush of dataSize ~78 B/78, heapSize ~472 B/472, currentSize=0 B/0 for 568ad5dacf91067899b20069681f9127 in 144ms, sequenceid=6, compaction requested=false at 1676077255024 (+18 ms)Writing region close event to WAL at 1676077255035 (+11 ms)Running coprocessor post-close hooks at 1676077255049 (+14 ms)Closed at 1676077255049 2023-02-11 01:00:55,050 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1676077253916.568ad5dacf91067899b20069681f9127. 2023-02-11 01:00:55,058 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44295/user/jenkins/test-data/593564ae-939c-6914-503a-94843f3f7058/data/hbase/meta/1588230740/.tmp/info/eb1d64753c804c89bf9b0a25b0e90524 as hdfs://localhost:44295/user/jenkins/test-data/593564ae-939c-6914-503a-94843f3f7058/data/hbase/meta/1588230740/info/eb1d64753c804c89bf9b0a25b0e90524 2023-02-11 01:00:55,068 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:44295/user/jenkins/test-data/593564ae-939c-6914-503a-94843f3f7058/data/hbase/meta/1588230740/info/eb1d64753c804c89bf9b0a25b0e90524, entries=10, sequenceid=9, filesize=5.9 K 2023-02-11 01:00:55,070 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:44295/user/jenkins/test-data/593564ae-939c-6914-503a-94843f3f7058/data/hbase/meta/1588230740/.tmp/table/3dff8b881acc49558fa232e5fc413259 as hdfs://localhost:44295/user/jenkins/test-data/593564ae-939c-6914-503a-94843f3f7058/data/hbase/meta/1588230740/table/3dff8b881acc49558fa232e5fc413259 2023-02-11 01:00:55,080 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:44295/user/jenkins/test-data/593564ae-939c-6914-503a-94843f3f7058/data/hbase/meta/1588230740/table/3dff8b881acc49558fa232e5fc413259, entries=2, sequenceid=9, filesize=4.7 K 2023-02-11 01:00:55,082 DEBUG [RS:2;jenkins-hbase4:42073] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-02-11 01:00:55,082 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~1.26 KB/1290, heapSize ~2.61 KB/2672, currentSize=0 B/0 for 1588230740 in 202ms, sequenceid=9, compaction requested=false 2023-02-11 01:00:55,082 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:meta' 2023-02-11 01:00:55,091 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:44295/user/jenkins/test-data/593564ae-939c-6914-503a-94843f3f7058/data/hbase/meta/1588230740/recovered.edits/12.seqid, newMaxSeqId=12, maxSeqId=1 2023-02-11 01:00:55,092 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-02-11 01:00:55,093 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed hbase:meta,,1.1588230740 2023-02-11 01:00:55,093 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 1588230740: Waiting for close lock at 1676077254879Running coprocessor pre-close hooks at 1676077254879Disabling compacts and flushes for region at 1676077254879Disabling writes for close at 1676077254880 (+1 ms)Obtaining lock to block concurrent updates at 1676077254881 (+1 ms)Preparing flush snapshotting stores in 1588230740 at 1676077254881Finished memstore snapshotting hbase:meta,,1.1588230740, syncing WAL and waiting on mvcc, flushsize=dataSize=1290, getHeapSize=2912, getOffHeapSize=0, getCellsCount=12 at 1676077254881Flushing stores of hbase:meta,,1.1588230740 at 1676077254885 (+4 ms)Flushing 1588230740/info: creating writer at 1676077254885Flushing 1588230740/info: appending metadata at 1676077254918 (+33 ms)Flushing 1588230740/info: closing flushed file at 1676077254921 (+3 ms)Flushing 1588230740/table: creating writer at 1676077255002 (+81 ms)Flushing 1588230740/table: appending metadata at 1676077255019 (+17 ms)Flushing 1588230740/table: closing flushed file at 1676077255019Flushing 1588230740/info: reopening flushed file at 1676077255060 (+41 ms)Flushing 1588230740/table: reopening flushed file at 1676077255071 (+11 ms)Finished flush of dataSize ~1.26 KB/1290, heapSize ~2.61 KB/2672, currentSize=0 B/0 for 1588230740 in 202ms, sequenceid=9, compaction requested=false at 1676077255082 (+11 ms)Writing region close event to WAL at 1676077255087 (+5 ms)Running coprocessor post-close hooks at 1676077255092 (+5 ms)Closed at 1676077255093 (+1 ms) 2023-02-11 01:00:55,093 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-02-11 01:00:55,133 DEBUG [Listener at localhost/46601-EventThread] zookeeper.ZKWatcher(600): regionserver:41571-0x10040e51f9c0001, quorum=127.0.0.1:63347, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-11 01:00:55,133 INFO [RS:0;jenkins-hbase4:41571] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,41571,1676077251993; zookeeper connection closed. 2023-02-11 01:00:55,133 ERROR [Listener at localhost/46601-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@3fd92729 rejected from java.util.concurrent.ThreadPoolExecutor@400973e8[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-11 01:00:55,134 DEBUG [Listener at localhost/46601-EventThread] zookeeper.ZKWatcher(600): regionserver:41571-0x10040e51f9c0001, quorum=127.0.0.1:63347, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-11 01:00:55,134 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@3339be75] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@3339be75 2023-02-11 01:00:55,134 ERROR [Listener at localhost/46601-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@423cd172 rejected from java.util.concurrent.ThreadPoolExecutor@400973e8[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-11 01:00:55,282 INFO [RS:2;jenkins-hbase4:42073] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,42073,1676077252069; all regions closed. 2023-02-11 01:00:55,289 DEBUG [RS:2;jenkins-hbase4:42073] wal.AbstractFSWAL(932): Moved 1 WAL file(s) to /user/jenkins/test-data/593564ae-939c-6914-503a-94843f3f7058/oldWALs 2023-02-11 01:00:55,290 INFO [RS:2;jenkins-hbase4:42073] wal.AbstractFSWAL(935): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C42073%2C1676077252069.meta:.meta(num 1676077253653) 2023-02-11 01:00:55,296 DEBUG [RS:2;jenkins-hbase4:42073] wal.AbstractFSWAL(932): Moved 1 WAL file(s) to /user/jenkins/test-data/593564ae-939c-6914-503a-94843f3f7058/oldWALs 2023-02-11 01:00:55,296 INFO [RS:2;jenkins-hbase4:42073] wal.AbstractFSWAL(935): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C42073%2C1676077252069:(num 1676077253545) 2023-02-11 01:00:55,296 DEBUG [RS:2;jenkins-hbase4:42073] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-11 01:00:55,296 INFO [RS:2;jenkins-hbase4:42073] regionserver.LeaseManager(133): Closed leases 2023-02-11 01:00:55,296 INFO [RS:2;jenkins-hbase4:42073] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS, ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS] on shutdown 2023-02-11 01:00:55,296 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-02-11 01:00:55,297 INFO [RS:2;jenkins-hbase4:42073] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:42073 2023-02-11 01:00:55,299 DEBUG [Listener at localhost/46601-EventThread] zookeeper.ZKWatcher(600): regionserver:42073-0x10040e51f9c0003, quorum=127.0.0.1:63347, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,42073,1676077252069 2023-02-11 01:00:55,299 ERROR [Listener at localhost/46601-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@62467a9e rejected from java.util.concurrent.ThreadPoolExecutor@6679f20e[Shutting down, pool size = 1, active threads = 0, queued tasks = 0, completed tasks = 8] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-11 01:00:55,433 DEBUG [Listener at localhost/46601-EventThread] zookeeper.ZKWatcher(600): regionserver:42073-0x10040e51f9c0003, quorum=127.0.0.1:63347, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-11 01:00:55,433 INFO [RS:2;jenkins-hbase4:42073] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,42073,1676077252069; zookeeper connection closed. 2023-02-11 01:00:55,433 ERROR [Listener at localhost/46601-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@43d157dd rejected from java.util.concurrent.ThreadPoolExecutor@6679f20e[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 8] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-11 01:00:55,434 DEBUG [Listener at localhost/46601-EventThread] zookeeper.ZKWatcher(600): regionserver:42073-0x10040e51f9c0003, quorum=127.0.0.1:63347, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-11 01:00:55,434 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@4a1c1687] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@4a1c1687 2023-02-11 01:00:55,434 ERROR [Listener at localhost/46601-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@3b522e7d rejected from java.util.concurrent.ThreadPoolExecutor@6679f20e[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 8] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-11 01:00:55,533 DEBUG [Listener at localhost/46601-EventThread] zookeeper.ZKWatcher(600): master:33145-0x10040e51f9c0000, quorum=127.0.0.1:63347, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-11 01:00:55,533 INFO [M:0;jenkins-hbase4:33145] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,33145,1676077250844; zookeeper connection closed. 2023-02-11 01:00:55,534 ERROR [Listener at localhost/46601-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@da5ff77 rejected from java.util.concurrent.ThreadPoolExecutor@7bb6b5f6[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 24] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-11 01:00:55,534 DEBUG [Listener at localhost/46601-EventThread] zookeeper.ZKWatcher(600): master:33145-0x10040e51f9c0000, quorum=127.0.0.1:63347, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-11 01:00:55,534 ERROR [Listener at localhost/46601-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@c994d9f rejected from java.util.concurrent.ThreadPoolExecutor@7bb6b5f6[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 24] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-11 01:00:55,634 DEBUG [Listener at localhost/46601-EventThread] zookeeper.ZKWatcher(600): regionserver:44999-0x10040e51f9c0002, quorum=127.0.0.1:63347, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-11 01:00:55,634 INFO [RS:1;jenkins-hbase4:44999] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,44999,1676077252037; zookeeper connection closed. 2023-02-11 01:00:55,634 ERROR [Listener at localhost/46601-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@419d95c7 rejected from java.util.concurrent.ThreadPoolExecutor@4876a4bb[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 6] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-11 01:00:55,634 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@766eb5ac] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@766eb5ac 2023-02-11 01:00:55,634 DEBUG [Listener at localhost/46601-EventThread] zookeeper.ZKWatcher(600): regionserver:44999-0x10040e51f9c0002, quorum=127.0.0.1:63347, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-11 01:00:55,634 ERROR [Listener at localhost/46601-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@56f9054b rejected from java.util.concurrent.ThreadPoolExecutor@4876a4bb[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 6] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-11 01:00:55,635 INFO [Listener at localhost/46601] util.JVMClusterUtil(335): Shutdown of 1 master(s) and 3 regionserver(s) complete 2023-02-11 01:00:55,635 WARN [Listener at localhost/46601] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-02-11 01:00:55,638 INFO [Listener at localhost/46601] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-02-11 01:00:55,707 WARN [BP-62378105-172.31.14.131-1676077247778 heartbeating to localhost/127.0.0.1:44295] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-62378105-172.31.14.131-1676077247778 (Datanode Uuid cdf5a302-30b0-4180-97ad-ca2e2d791bac) service to localhost/127.0.0.1:44295 2023-02-11 01:00:55,709 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e5dd7502-721b-939f-9fad-68d9c52e98bf/cluster_57098650-f7a3-dd69-97b5-a6c2a3ffc93f/dfs/data/data5/current/BP-62378105-172.31.14.131-1676077247778] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-11 01:00:55,709 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e5dd7502-721b-939f-9fad-68d9c52e98bf/cluster_57098650-f7a3-dd69-97b5-a6c2a3ffc93f/dfs/data/data6/current/BP-62378105-172.31.14.131-1676077247778] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-11 01:00:55,740 WARN [Listener at localhost/46601] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-02-11 01:00:55,742 INFO [Listener at localhost/46601] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-02-11 01:00:55,845 WARN [BP-62378105-172.31.14.131-1676077247778 heartbeating to localhost/127.0.0.1:44295] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-02-11 01:00:55,845 WARN [BP-62378105-172.31.14.131-1676077247778 heartbeating to localhost/127.0.0.1:44295] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-62378105-172.31.14.131-1676077247778 (Datanode Uuid 5d966976-4df8-4674-b9ce-a32041365514) service to localhost/127.0.0.1:44295 2023-02-11 01:00:55,845 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e5dd7502-721b-939f-9fad-68d9c52e98bf/cluster_57098650-f7a3-dd69-97b5-a6c2a3ffc93f/dfs/data/data3/current/BP-62378105-172.31.14.131-1676077247778] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-11 01:00:55,846 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e5dd7502-721b-939f-9fad-68d9c52e98bf/cluster_57098650-f7a3-dd69-97b5-a6c2a3ffc93f/dfs/data/data4/current/BP-62378105-172.31.14.131-1676077247778] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-11 01:00:55,846 WARN [Listener at localhost/46601] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-02-11 01:00:55,848 INFO [Listener at localhost/46601] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-02-11 01:00:55,950 WARN [BP-62378105-172.31.14.131-1676077247778 heartbeating to localhost/127.0.0.1:44295] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-02-11 01:00:55,950 WARN [BP-62378105-172.31.14.131-1676077247778 heartbeating to localhost/127.0.0.1:44295] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-62378105-172.31.14.131-1676077247778 (Datanode Uuid 0ed6119c-6e7c-46d2-bdd7-f9fb0be3812e) service to localhost/127.0.0.1:44295 2023-02-11 01:00:55,951 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e5dd7502-721b-939f-9fad-68d9c52e98bf/cluster_57098650-f7a3-dd69-97b5-a6c2a3ffc93f/dfs/data/data1/current/BP-62378105-172.31.14.131-1676077247778] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-11 01:00:55,951 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e5dd7502-721b-939f-9fad-68d9c52e98bf/cluster_57098650-f7a3-dd69-97b5-a6c2a3ffc93f/dfs/data/data2/current/BP-62378105-172.31.14.131-1676077247778] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-11 01:00:55,977 INFO [Listener at localhost/46601] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-02-11 01:00:56,091 INFO [Listener at localhost/46601] zookeeper.MiniZooKeeperCluster(344): Shutdown MiniZK cluster with all ZK servers 2023-02-11 01:00:56,129 INFO [Listener at localhost/46601] hbase.HBaseTestingUtility(1293): Minicluster is down 2023-02-11 01:00:56,141 INFO [Listener at localhost/46601] hbase.ResourceChecker(175): after: client.TestAsyncClusterAdminApi2#testStop Thread=77 (was 8) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-3 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: regionserver/jenkins-hbase4:0.procedureResultReporter sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.regionserver.RemoteProcedureResultReporter.run(RemoteProcedureResultReporter.java:77) Potentially hanging thread: org.apache.hadoop.hdfs.PeerCache@d1663db java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.PeerCache.run(PeerCache.java:253) org.apache.hadoop.hdfs.PeerCache.access$000(PeerCache.java:46) org.apache.hadoop.hdfs.PeerCache$1.run(PeerCache.java:124) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: IPC Client (762418739) connection to localhost/127.0.0.1:44295 from jenkins.hfs.1 java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1035) org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079) Potentially hanging thread: RS-EventLoopGroup-5-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-2 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-1-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: regionserver/jenkins-hbase4:0.procedureResultReporter sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.regionserver.RemoteProcedureResultReporter.run(RemoteProcedureResultReporter.java:77) Potentially hanging thread: ReplicationExecutor-0 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager$NodeFailoverWorker.run(ReplicationSourceManager.java:703) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ForkJoinPool-2-worker-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.ForkJoinPool.awaitWork(ForkJoinPool.java:1824) java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1693) java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) Potentially hanging thread: ForkJoinPool-2-worker-5 sun.misc.Unsafe.park(Native Method) java.util.concurrent.ForkJoinPool.awaitWork(ForkJoinPool.java:1824) java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1693) java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) Potentially hanging thread: IPC Client (762418739) connection to localhost/127.0.0.1:44295 from jenkins.hfs.0 java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1035) org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079) Potentially hanging thread: nioEventLoopGroup-3-3 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RpcClient-timer-pool-0 java.lang.Thread.sleep(Native Method) org.apache.hbase.thirdparty.io.netty.util.HashedWheelTimer$Worker.waitForNextTick(HashedWheelTimer.java:600) org.apache.hbase.thirdparty.io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:496) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReplicationExecutor-0 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager$NodeFailoverWorker.run(ReplicationSourceManager.java:703) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-5-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: IPC Parameter Sending Thread #2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: region-location-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-7-1 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-1-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Monitor thread for TaskMonitor java.lang.Thread.sleep(Native Method) org.apache.hadoop.hbase.monitoring.TaskMonitor$MonitorRunnable.run(TaskMonitor.java:300) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: region-location-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-2-2 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-2-1 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HBase-Metrics2-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: regionserver/jenkins-hbase4:0.procedureResultReporter sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.hadoop.hbase.regionserver.RemoteProcedureResultReporter.run(RemoteProcedureResultReporter.java:77) Potentially hanging thread: IPC Parameter Sending Thread #0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-2-3 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-3-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-3-1 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-4-1 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-1-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: SessionTracker java.lang.Thread.sleep(Native Method) org.apache.zookeeper.server.SessionTrackerImpl.run(SessionTrackerImpl.java:151) Potentially hanging thread: RS-EventLoopGroup-4-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: IPC Client (762418739) connection to localhost/127.0.0.1:44295 from jenkins java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1035) org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079) Potentially hanging thread: IPC Client (762418739) connection to localhost/127.0.0.1:44295 from jenkins java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1035) org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079) Potentially hanging thread: nioEventLoopGroup-5-2 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner java.lang.Object.wait(Native Method) java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:144) java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:165) org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner.run(FileSystem.java:3693) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-4-2 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Idle-Rpc-Conn-Sweeper-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ForkJoinPool-2-worker-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.ForkJoinPool.awaitWork(ForkJoinPool.java:1824) java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1693) java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) Potentially hanging thread: master/jenkins-hbase4:0:becomeActiveMaster-MemStoreChunkPool Statistics sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-4-3 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ForkJoinPool-2-worker-4 sun.misc.Unsafe.park(Native Method) java.util.concurrent.ForkJoinPool.awaitWork(ForkJoinPool.java:1824) java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1693) java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) Potentially hanging thread: nioEventLoopGroup-6-3 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-5-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-5-3 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Timer for 'NameNode' metrics system java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: ForkJoinPool-2-worker-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.ForkJoinPool.awaitWork(ForkJoinPool.java:1824) java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1693) java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) Potentially hanging thread: LeaseRenewer:jenkins.hfs.2@localhost:44295 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.run(LeaseRenewer.java:411) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.access$600(LeaseRenewer.java:76) org.apache.hadoop.hdfs.client.impl.LeaseRenewer$1.run(LeaseRenewer.java:307) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-3-2 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Listener at localhost/46601 java.lang.Thread.dumpThreads(Native Method) java.lang.Thread.getAllStackTraces(Thread.java:1615) org.apache.hadoop.hbase.ResourceCheckerJUnitListener$ThreadResourceAnalyzer.getVal(ResourceCheckerJUnitListener.java:49) org.apache.hadoop.hbase.ResourceChecker.fill(ResourceChecker.java:110) org.apache.hadoop.hbase.ResourceChecker.fillEndings(ResourceChecker.java:104) org.apache.hadoop.hbase.ResourceChecker.end(ResourceChecker.java:206) org.apache.hadoop.hbase.ResourceCheckerJUnitListener.end(ResourceCheckerJUnitListener.java:165) org.apache.hadoop.hbase.ResourceCheckerJUnitListener.testFinished(ResourceCheckerJUnitListener.java:185) org.junit.runner.notification.SynchronizedRunListener.testFinished(SynchronizedRunListener.java:87) org.junit.runner.notification.RunNotifier$9.notifyListener(RunNotifier.java:225) org.junit.runner.notification.RunNotifier$SafeNotifier.run(RunNotifier.java:72) org.junit.runner.notification.RunNotifier.fireTestFinished(RunNotifier.java:222) org.junit.internal.runners.model.EachTestNotifier.fireTestFinished(EachTestNotifier.java:38) org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:372) org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103) org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63) org.junit.runners.ParentRunner$4.run(ParentRunner.java:331) org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79) org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329) org.junit.runners.ParentRunner.access$100(ParentRunner.java:66) org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293) org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) org.apache.hadoop.hbase.SystemExitRule$1.evaluate(SystemExitRule.java:39) org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299) org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293) java.util.concurrent.FutureTask.run(FutureTask.java:266) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: IPC Parameter Sending Thread #1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: LeaseRenewer:jenkins.hfs.1@localhost:44295 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.run(LeaseRenewer.java:411) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.access$600(LeaseRenewer.java:76) org.apache.hadoop.hdfs.client.impl.LeaseRenewer$1.run(LeaseRenewer.java:307) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-7-3 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-6-2 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: LeaseRenewer:jenkins.hfs.0@localhost:44295 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.run(LeaseRenewer.java:411) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.access$600(LeaseRenewer.java:76) org.apache.hadoop.hdfs.client.impl.LeaseRenewer$1.run(LeaseRenewer.java:307) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-5-1 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: master/jenkins-hbase4:0:becomeActiveMaster-MemStoreChunkPool Statistics sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-7-2 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: SnapshotHandlerChoreCleaner sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: LeaseRenewer:jenkins@localhost:44295 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.run(LeaseRenewer.java:411) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.access$600(LeaseRenewer.java:76) org.apache.hadoop.hdfs.client.impl.LeaseRenewer$1.run(LeaseRenewer.java:307) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-3-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-6-1 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: IPC Client (762418739) connection to localhost/127.0.0.1:44295 from jenkins.hfs.2 java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1035) org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-1 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) - Thread LEAK? -, OpenFileDescriptor=497 (was 260) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=186 (was 184) - SystemLoadAverage LEAK? -, ProcessCount=168 (was 168), AvailableMemoryMB=3171 (was 3766) 2023-02-11 01:00:56,151 INFO [Listener at localhost/46601] hbase.ResourceChecker(147): before: client.TestAsyncClusterAdminApi2#testShutdown Thread=77, OpenFileDescriptor=497, MaxFileDescriptor=60000, SystemLoadAverage=186, ProcessCount=168, AvailableMemoryMB=3171 2023-02-11 01:00:56,151 INFO [Listener at localhost/46601] hbase.HBaseTestingUtility(1068): Starting up minicluster with option: StartMiniClusterOption{numMasters=1, masterClass=null, numRegionServers=3, rsPorts=, rsClass=null, numDataNodes=3, dataNodeHosts=null, numZkServers=1, createRootDir=false, createWALDir=false} 2023-02-11 01:00:56,151 INFO [Listener at localhost/46601] hbase.HBaseTestingUtility(445): System.getProperty("hadoop.log.dir") already set to: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e5dd7502-721b-939f-9fad-68d9c52e98bf/hadoop.log.dir so I do NOT create it in target/test-data/a0aedee6-e991-0fa5-31da-b7ab416d8fdd 2023-02-11 01:00:56,151 INFO [Listener at localhost/46601] hbase.HBaseTestingUtility(445): System.getProperty("hadoop.tmp.dir") already set to: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e5dd7502-721b-939f-9fad-68d9c52e98bf/hadoop.tmp.dir so I do NOT create it in target/test-data/a0aedee6-e991-0fa5-31da-b7ab416d8fdd 2023-02-11 01:00:56,151 INFO [Listener at localhost/46601] hbase.HBaseZKTestingUtility(82): Created new mini-cluster data directory: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a0aedee6-e991-0fa5-31da-b7ab416d8fdd/cluster_f0858fb3-f374-2678-20e0-92155084cac0, deleteOnExit=true 2023-02-11 01:00:56,151 INFO [Listener at localhost/46601] hbase.HBaseTestingUtility(1082): STARTING DFS 2023-02-11 01:00:56,152 INFO [Listener at localhost/46601] hbase.HBaseTestingUtility(772): Setting test.cache.data to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a0aedee6-e991-0fa5-31da-b7ab416d8fdd/test.cache.data in system properties and HBase conf 2023-02-11 01:00:56,152 INFO [Listener at localhost/46601] hbase.HBaseTestingUtility(772): Setting hadoop.tmp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a0aedee6-e991-0fa5-31da-b7ab416d8fdd/hadoop.tmp.dir in system properties and HBase conf 2023-02-11 01:00:56,152 INFO [Listener at localhost/46601] hbase.HBaseTestingUtility(772): Setting hadoop.log.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a0aedee6-e991-0fa5-31da-b7ab416d8fdd/hadoop.log.dir in system properties and HBase conf 2023-02-11 01:00:56,152 INFO [Listener at localhost/46601] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.local.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a0aedee6-e991-0fa5-31da-b7ab416d8fdd/mapreduce.cluster.local.dir in system properties and HBase conf 2023-02-11 01:00:56,152 INFO [Listener at localhost/46601] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.temp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a0aedee6-e991-0fa5-31da-b7ab416d8fdd/mapreduce.cluster.temp.dir in system properties and HBase conf 2023-02-11 01:00:56,152 INFO [Listener at localhost/46601] hbase.HBaseTestingUtility(759): read short circuit is OFF 2023-02-11 01:00:56,153 DEBUG [Listener at localhost/46601] fs.HFileSystem(308): The file system is not a DistributedFileSystem. Skipping on block location reordering 2023-02-11 01:00:56,153 INFO [Listener at localhost/46601] hbase.HBaseTestingUtility(772): Setting yarn.node-labels.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a0aedee6-e991-0fa5-31da-b7ab416d8fdd/yarn.node-labels.fs-store.root-dir in system properties and HBase conf 2023-02-11 01:00:56,153 INFO [Listener at localhost/46601] hbase.HBaseTestingUtility(772): Setting yarn.node-attribute.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a0aedee6-e991-0fa5-31da-b7ab416d8fdd/yarn.node-attribute.fs-store.root-dir in system properties and HBase conf 2023-02-11 01:00:56,153 INFO [Listener at localhost/46601] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.log-dirs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a0aedee6-e991-0fa5-31da-b7ab416d8fdd/yarn.nodemanager.log-dirs in system properties and HBase conf 2023-02-11 01:00:56,153 INFO [Listener at localhost/46601] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a0aedee6-e991-0fa5-31da-b7ab416d8fdd/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-02-11 01:00:56,154 INFO [Listener at localhost/46601] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.active-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a0aedee6-e991-0fa5-31da-b7ab416d8fdd/yarn.timeline-service.entity-group-fs-store.active-dir in system properties and HBase conf 2023-02-11 01:00:56,154 INFO [Listener at localhost/46601] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.done-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a0aedee6-e991-0fa5-31da-b7ab416d8fdd/yarn.timeline-service.entity-group-fs-store.done-dir in system properties and HBase conf 2023-02-11 01:00:56,154 INFO [Listener at localhost/46601] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a0aedee6-e991-0fa5-31da-b7ab416d8fdd/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-02-11 01:00:56,154 INFO [Listener at localhost/46601] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a0aedee6-e991-0fa5-31da-b7ab416d8fdd/dfs.journalnode.edits.dir in system properties and HBase conf 2023-02-11 01:00:56,154 INFO [Listener at localhost/46601] hbase.HBaseTestingUtility(772): Setting dfs.datanode.shared.file.descriptor.paths to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a0aedee6-e991-0fa5-31da-b7ab416d8fdd/dfs.datanode.shared.file.descriptor.paths in system properties and HBase conf 2023-02-11 01:00:56,154 INFO [Listener at localhost/46601] hbase.HBaseTestingUtility(772): Setting nfs.dump.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a0aedee6-e991-0fa5-31da-b7ab416d8fdd/nfs.dump.dir in system properties and HBase conf 2023-02-11 01:00:56,154 INFO [Listener at localhost/46601] hbase.HBaseTestingUtility(772): Setting java.io.tmpdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a0aedee6-e991-0fa5-31da-b7ab416d8fdd/java.io.tmpdir in system properties and HBase conf 2023-02-11 01:00:56,154 INFO [Listener at localhost/46601] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a0aedee6-e991-0fa5-31da-b7ab416d8fdd/dfs.journalnode.edits.dir in system properties and HBase conf 2023-02-11 01:00:56,154 INFO [Listener at localhost/46601] hbase.HBaseTestingUtility(772): Setting dfs.provided.aliasmap.inmemory.leveldb.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a0aedee6-e991-0fa5-31da-b7ab416d8fdd/dfs.provided.aliasmap.inmemory.leveldb.dir in system properties and HBase conf 2023-02-11 01:00:56,155 INFO [Listener at localhost/46601] hbase.HBaseTestingUtility(772): Setting fs.s3a.committer.staging.tmp.path to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a0aedee6-e991-0fa5-31da-b7ab416d8fdd/fs.s3a.committer.staging.tmp.path in system properties and HBase conf Formatting using clusterid: testClusterID 2023-02-11 01:00:56,159 WARN [Listener at localhost/46601] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-02-11 01:00:56,159 WARN [Listener at localhost/46601] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-02-11 01:00:56,200 WARN [Listener at localhost/46601] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-02-11 01:00:56,203 INFO [Listener at localhost/46601] log.Slf4jLog(67): jetty-6.1.26 2023-02-11 01:00:56,211 INFO [Listener at localhost/46601] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/hdfs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a0aedee6-e991-0fa5-31da-b7ab416d8fdd/java.io.tmpdir/Jetty_localhost_35165_hdfs____3gpu48/webapp 2023-02-11 01:00:56,309 INFO [Listener at localhost/46601] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:35165 2023-02-11 01:00:56,314 WARN [Listener at localhost/46601] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-02-11 01:00:56,314 WARN [Listener at localhost/46601] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-02-11 01:00:56,355 WARN [Listener at localhost/45625] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-02-11 01:00:56,366 WARN [Listener at localhost/45625] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-02-11 01:00:56,368 WARN [Listener at localhost/45625] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-02-11 01:00:56,369 INFO [Listener at localhost/45625] log.Slf4jLog(67): jetty-6.1.26 2023-02-11 01:00:56,374 INFO [Listener at localhost/45625] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a0aedee6-e991-0fa5-31da-b7ab416d8fdd/java.io.tmpdir/Jetty_localhost_43231_datanode____7ql9co/webapp 2023-02-11 01:00:56,465 INFO [Listener at localhost/45625] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:43231 2023-02-11 01:00:56,474 WARN [Listener at localhost/36545] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-02-11 01:00:56,493 WARN [Listener at localhost/36545] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-02-11 01:00:56,496 WARN [Listener at localhost/36545] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-02-11 01:00:56,497 INFO [Listener at localhost/36545] log.Slf4jLog(67): jetty-6.1.26 2023-02-11 01:00:56,500 INFO [Listener at localhost/36545] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a0aedee6-e991-0fa5-31da-b7ab416d8fdd/java.io.tmpdir/Jetty_localhost_44571_datanode____.udsm6i/webapp 2023-02-11 01:00:56,582 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x5ae8aa8a7431bd45: Processing first storage report for DS-2d423c52-201c-4032-9b49-b39dee42b6b7 from datanode 0a3f943a-4e83-433c-8383-2ecc4ab4aae9 2023-02-11 01:00:56,583 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x5ae8aa8a7431bd45: from storage DS-2d423c52-201c-4032-9b49-b39dee42b6b7 node DatanodeRegistration(127.0.0.1:43101, datanodeUuid=0a3f943a-4e83-433c-8383-2ecc4ab4aae9, infoPort=36171, infoSecurePort=0, ipcPort=36545, storageInfo=lv=-57;cid=testClusterID;nsid=430685456;c=1676077256162), blocks: 0, hasStaleStorage: true, processing time: 1 msecs, invalidatedBlocks: 0 2023-02-11 01:00:56,583 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x5ae8aa8a7431bd45: Processing first storage report for DS-ff033b51-b06e-4624-a1b7-637e243683b0 from datanode 0a3f943a-4e83-433c-8383-2ecc4ab4aae9 2023-02-11 01:00:56,583 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x5ae8aa8a7431bd45: from storage DS-ff033b51-b06e-4624-a1b7-637e243683b0 node DatanodeRegistration(127.0.0.1:43101, datanodeUuid=0a3f943a-4e83-433c-8383-2ecc4ab4aae9, infoPort=36171, infoSecurePort=0, ipcPort=36545, storageInfo=lv=-57;cid=testClusterID;nsid=430685456;c=1676077256162), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-02-11 01:00:56,612 INFO [Listener at localhost/36545] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:44571 2023-02-11 01:00:56,620 WARN [Listener at localhost/38381] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-02-11 01:00:56,633 WARN [Listener at localhost/38381] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-02-11 01:00:56,635 WARN [Listener at localhost/38381] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-02-11 01:00:56,636 INFO [Listener at localhost/38381] log.Slf4jLog(67): jetty-6.1.26 2023-02-11 01:00:56,640 INFO [Listener at localhost/38381] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a0aedee6-e991-0fa5-31da-b7ab416d8fdd/java.io.tmpdir/Jetty_localhost_38147_datanode____msmu6u/webapp 2023-02-11 01:00:56,712 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xb601af23cdd7f00f: Processing first storage report for DS-2350da3f-1592-4ca5-a29c-11cf6e912f79 from datanode 4fad92d9-c302-4bf9-98ed-2c4130b68882 2023-02-11 01:00:56,712 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xb601af23cdd7f00f: from storage DS-2350da3f-1592-4ca5-a29c-11cf6e912f79 node DatanodeRegistration(127.0.0.1:35587, datanodeUuid=4fad92d9-c302-4bf9-98ed-2c4130b68882, infoPort=40493, infoSecurePort=0, ipcPort=38381, storageInfo=lv=-57;cid=testClusterID;nsid=430685456;c=1676077256162), blocks: 0, hasStaleStorage: true, processing time: 0 msecs, invalidatedBlocks: 0 2023-02-11 01:00:56,712 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xb601af23cdd7f00f: Processing first storage report for DS-66f6f746-86cf-43dd-b1a4-6e7fbf5f420d from datanode 4fad92d9-c302-4bf9-98ed-2c4130b68882 2023-02-11 01:00:56,712 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xb601af23cdd7f00f: from storage DS-66f6f746-86cf-43dd-b1a4-6e7fbf5f420d node DatanodeRegistration(127.0.0.1:35587, datanodeUuid=4fad92d9-c302-4bf9-98ed-2c4130b68882, infoPort=40493, infoSecurePort=0, ipcPort=38381, storageInfo=lv=-57;cid=testClusterID;nsid=430685456;c=1676077256162), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-02-11 01:00:56,757 INFO [Listener at localhost/38381] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:38147 2023-02-11 01:00:56,765 WARN [Listener at localhost/43805] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-02-11 01:00:56,862 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xae81d3f69393bf00: Processing first storage report for DS-30862ee3-36ca-4a21-884e-547ca76a2efc from datanode b724d071-d6c0-4206-8e44-df01650df727 2023-02-11 01:00:56,862 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xae81d3f69393bf00: from storage DS-30862ee3-36ca-4a21-884e-547ca76a2efc node DatanodeRegistration(127.0.0.1:38851, datanodeUuid=b724d071-d6c0-4206-8e44-df01650df727, infoPort=39415, infoSecurePort=0, ipcPort=43805, storageInfo=lv=-57;cid=testClusterID;nsid=430685456;c=1676077256162), blocks: 0, hasStaleStorage: true, processing time: 1 msecs, invalidatedBlocks: 0 2023-02-11 01:00:56,862 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xae81d3f69393bf00: Processing first storage report for DS-e51c63a7-5eec-4739-909c-3fb2a2929e7c from datanode b724d071-d6c0-4206-8e44-df01650df727 2023-02-11 01:00:56,862 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xae81d3f69393bf00: from storage DS-e51c63a7-5eec-4739-909c-3fb2a2929e7c node DatanodeRegistration(127.0.0.1:38851, datanodeUuid=b724d071-d6c0-4206-8e44-df01650df727, infoPort=39415, infoSecurePort=0, ipcPort=43805, storageInfo=lv=-57;cid=testClusterID;nsid=430685456;c=1676077256162), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-02-11 01:00:56,873 DEBUG [Listener at localhost/43805] hbase.HBaseTestingUtility(649): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a0aedee6-e991-0fa5-31da-b7ab416d8fdd 2023-02-11 01:00:56,876 INFO [Listener at localhost/43805] zookeeper.MiniZooKeeperCluster(258): Started connectionTimeout=30000, dir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a0aedee6-e991-0fa5-31da-b7ab416d8fdd/cluster_f0858fb3-f374-2678-20e0-92155084cac0/zookeeper_0, clientPort=55753, secureClientPort=-1, dataDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a0aedee6-e991-0fa5-31da-b7ab416d8fdd/cluster_f0858fb3-f374-2678-20e0-92155084cac0/zookeeper_0/version-2, dataDirSize=424 dataLogDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a0aedee6-e991-0fa5-31da-b7ab416d8fdd/cluster_f0858fb3-f374-2678-20e0-92155084cac0/zookeeper_0/version-2, dataLogSize=424 tickTime=2000, maxClientCnxns=300, minSessionTimeout=4000, maxSessionTimeout=40000, serverId=0 2023-02-11 01:00:56,878 INFO [Listener at localhost/43805] zookeeper.MiniZooKeeperCluster(283): Started MiniZooKeeperCluster and ran 'stat' on client port=55753 2023-02-11 01:00:56,878 INFO [Listener at localhost/43805] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-11 01:00:56,879 INFO [Listener at localhost/43805] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-11 01:00:56,906 INFO [Listener at localhost/43805] util.FSUtils(479): Created version file at hdfs://localhost:45625/user/jenkins/test-data/2a140438-d2b1-6994-16d5-13e6c7986734 with version=8 2023-02-11 01:00:56,906 INFO [Listener at localhost/43805] hbase.HBaseTestingUtility(1408): The hbase.fs.tmp.dir is set to hdfs://localhost:44295/user/jenkins/test-data/593564ae-939c-6914-503a-94843f3f7058/hbase-staging 2023-02-11 01:00:56,908 INFO [Listener at localhost/43805] client.ConnectionUtils(127): master/jenkins-hbase4:0 server-side Connection retries=6 2023-02-11 01:00:56,908 INFO [Listener at localhost/43805] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-11 01:00:56,909 INFO [Listener at localhost/43805] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-02-11 01:00:56,909 INFO [Listener at localhost/43805] ipc.RWQueueRpcExecutor(105): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-02-11 01:00:56,909 INFO [Listener at localhost/43805] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-11 01:00:56,909 INFO [Listener at localhost/43805] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-02-11 01:00:56,909 INFO [Listener at localhost/43805] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-02-11 01:00:56,911 INFO [Listener at localhost/43805] ipc.NettyRpcServer(120): Bind to /172.31.14.131:46619 2023-02-11 01:00:56,911 INFO [Listener at localhost/43805] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-11 01:00:56,912 INFO [Listener at localhost/43805] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-11 01:00:56,913 INFO [Listener at localhost/43805] zookeeper.RecoverableZooKeeper(93): Process identifier=master:46619 connecting to ZooKeeper ensemble=127.0.0.1:55753 2023-02-11 01:00:56,920 DEBUG [Listener at localhost/43805-EventThread] zookeeper.ZKWatcher(600): master:466190x0, quorum=127.0.0.1:55753, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-02-11 01:00:56,922 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): master:46619-0x10040e53a700000 connected 2023-02-11 01:00:56,940 DEBUG [Listener at localhost/43805] zookeeper.ZKUtil(164): master:46619-0x10040e53a700000, quorum=127.0.0.1:55753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-11 01:00:56,940 DEBUG [Listener at localhost/43805] zookeeper.ZKUtil(164): master:46619-0x10040e53a700000, quorum=127.0.0.1:55753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-11 01:00:56,941 DEBUG [Listener at localhost/43805] zookeeper.ZKUtil(164): master:46619-0x10040e53a700000, quorum=127.0.0.1:55753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-02-11 01:00:56,941 DEBUG [Listener at localhost/43805] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=46619 2023-02-11 01:00:56,941 DEBUG [Listener at localhost/43805] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=46619 2023-02-11 01:00:56,941 DEBUG [Listener at localhost/43805] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=46619 2023-02-11 01:00:56,942 DEBUG [Listener at localhost/43805] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=46619 2023-02-11 01:00:56,942 DEBUG [Listener at localhost/43805] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=46619 2023-02-11 01:00:56,942 INFO [Listener at localhost/43805] master.HMaster(439): hbase.rootdir=hdfs://localhost:45625/user/jenkins/test-data/2a140438-d2b1-6994-16d5-13e6c7986734, hbase.cluster.distributed=false 2023-02-11 01:00:56,956 INFO [Listener at localhost/43805] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=6 2023-02-11 01:00:56,956 INFO [Listener at localhost/43805] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-11 01:00:56,956 INFO [Listener at localhost/43805] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-02-11 01:00:56,956 INFO [Listener at localhost/43805] ipc.RWQueueRpcExecutor(105): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-02-11 01:00:56,956 INFO [Listener at localhost/43805] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-11 01:00:56,957 INFO [Listener at localhost/43805] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-02-11 01:00:56,957 INFO [Listener at localhost/43805] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-02-11 01:00:56,958 INFO [Listener at localhost/43805] ipc.NettyRpcServer(120): Bind to /172.31.14.131:36543 2023-02-11 01:00:56,958 INFO [Listener at localhost/43805] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-02-11 01:00:56,959 DEBUG [Listener at localhost/43805] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-02-11 01:00:56,960 INFO [Listener at localhost/43805] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-11 01:00:56,961 INFO [Listener at localhost/43805] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-11 01:00:56,962 INFO [Listener at localhost/43805] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:36543 connecting to ZooKeeper ensemble=127.0.0.1:55753 2023-02-11 01:00:56,965 DEBUG [Listener at localhost/43805-EventThread] zookeeper.ZKWatcher(600): regionserver:365430x0, quorum=127.0.0.1:55753, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-02-11 01:00:56,966 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:36543-0x10040e53a700001 connected 2023-02-11 01:00:56,967 DEBUG [Listener at localhost/43805] zookeeper.ZKUtil(164): regionserver:36543-0x10040e53a700001, quorum=127.0.0.1:55753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-11 01:00:56,967 DEBUG [Listener at localhost/43805] zookeeper.ZKUtil(164): regionserver:36543-0x10040e53a700001, quorum=127.0.0.1:55753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-11 01:00:56,968 DEBUG [Listener at localhost/43805] zookeeper.ZKUtil(164): regionserver:36543-0x10040e53a700001, quorum=127.0.0.1:55753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-02-11 01:00:56,972 DEBUG [Listener at localhost/43805] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=36543 2023-02-11 01:00:56,972 DEBUG [Listener at localhost/43805] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=36543 2023-02-11 01:00:56,972 DEBUG [Listener at localhost/43805] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=36543 2023-02-11 01:00:56,974 DEBUG [Listener at localhost/43805] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=36543 2023-02-11 01:00:56,974 DEBUG [Listener at localhost/43805] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=36543 2023-02-11 01:00:56,985 INFO [Listener at localhost/43805] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=6 2023-02-11 01:00:56,985 INFO [Listener at localhost/43805] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-11 01:00:56,986 INFO [Listener at localhost/43805] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-02-11 01:00:56,986 INFO [Listener at localhost/43805] ipc.RWQueueRpcExecutor(105): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-02-11 01:00:56,986 INFO [Listener at localhost/43805] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-11 01:00:56,986 INFO [Listener at localhost/43805] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-02-11 01:00:56,986 INFO [Listener at localhost/43805] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-02-11 01:00:56,987 INFO [Listener at localhost/43805] ipc.NettyRpcServer(120): Bind to /172.31.14.131:38923 2023-02-11 01:00:56,988 INFO [Listener at localhost/43805] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-02-11 01:00:56,989 DEBUG [Listener at localhost/43805] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-02-11 01:00:56,989 INFO [Listener at localhost/43805] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-11 01:00:56,990 INFO [Listener at localhost/43805] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-11 01:00:56,992 INFO [Listener at localhost/43805] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:38923 connecting to ZooKeeper ensemble=127.0.0.1:55753 2023-02-11 01:00:56,998 DEBUG [Listener at localhost/43805-EventThread] zookeeper.ZKWatcher(600): regionserver:389230x0, quorum=127.0.0.1:55753, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-02-11 01:00:56,999 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:38923-0x10040e53a700002 connected 2023-02-11 01:00:56,999 DEBUG [Listener at localhost/43805] zookeeper.ZKUtil(164): regionserver:38923-0x10040e53a700002, quorum=127.0.0.1:55753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-11 01:00:57,000 DEBUG [Listener at localhost/43805] zookeeper.ZKUtil(164): regionserver:38923-0x10040e53a700002, quorum=127.0.0.1:55753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-11 01:00:57,000 DEBUG [Listener at localhost/43805] zookeeper.ZKUtil(164): regionserver:38923-0x10040e53a700002, quorum=127.0.0.1:55753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-02-11 01:00:57,002 DEBUG [Listener at localhost/43805] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=38923 2023-02-11 01:00:57,003 DEBUG [Listener at localhost/43805] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=38923 2023-02-11 01:00:57,003 DEBUG [Listener at localhost/43805] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=38923 2023-02-11 01:00:57,004 DEBUG [Listener at localhost/43805] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=38923 2023-02-11 01:00:57,008 DEBUG [Listener at localhost/43805] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=38923 2023-02-11 01:00:57,019 INFO [Listener at localhost/43805] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=6 2023-02-11 01:00:57,020 INFO [Listener at localhost/43805] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-11 01:00:57,020 INFO [Listener at localhost/43805] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-02-11 01:00:57,020 INFO [Listener at localhost/43805] ipc.RWQueueRpcExecutor(105): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-02-11 01:00:57,020 INFO [Listener at localhost/43805] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-11 01:00:57,020 INFO [Listener at localhost/43805] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-02-11 01:00:57,020 INFO [Listener at localhost/43805] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-02-11 01:00:57,028 INFO [Listener at localhost/43805] ipc.NettyRpcServer(120): Bind to /172.31.14.131:43507 2023-02-11 01:00:57,029 INFO [Listener at localhost/43805] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-02-11 01:00:57,034 DEBUG [Listener at localhost/43805] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-02-11 01:00:57,034 INFO [Listener at localhost/43805] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-11 01:00:57,036 INFO [Listener at localhost/43805] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-11 01:00:57,037 INFO [Listener at localhost/43805] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:43507 connecting to ZooKeeper ensemble=127.0.0.1:55753 2023-02-11 01:00:57,042 DEBUG [Listener at localhost/43805-EventThread] zookeeper.ZKWatcher(600): regionserver:435070x0, quorum=127.0.0.1:55753, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-02-11 01:00:57,043 DEBUG [Listener at localhost/43805] zookeeper.ZKUtil(164): regionserver:435070x0, quorum=127.0.0.1:55753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-11 01:00:57,043 DEBUG [Listener at localhost/43805] zookeeper.ZKUtil(164): regionserver:435070x0, quorum=127.0.0.1:55753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-11 01:00:57,044 DEBUG [Listener at localhost/43805] zookeeper.ZKUtil(164): regionserver:435070x0, quorum=127.0.0.1:55753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-02-11 01:00:57,044 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:43507-0x10040e53a700003 connected 2023-02-11 01:00:57,045 DEBUG [Listener at localhost/43805] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=43507 2023-02-11 01:00:57,045 DEBUG [Listener at localhost/43805] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=43507 2023-02-11 01:00:57,045 DEBUG [Listener at localhost/43805] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=43507 2023-02-11 01:00:57,046 DEBUG [Listener at localhost/43805] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=43507 2023-02-11 01:00:57,046 DEBUG [Listener at localhost/43805] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=43507 2023-02-11 01:00:57,047 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2158): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase4.apache.org,46619,1676077256908 2023-02-11 01:00:57,049 DEBUG [Listener at localhost/43805-EventThread] zookeeper.ZKWatcher(600): master:46619-0x10040e53a700000, quorum=127.0.0.1:55753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-02-11 01:00:57,050 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:46619-0x10040e53a700000, quorum=127.0.0.1:55753, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase4.apache.org,46619,1676077256908 2023-02-11 01:00:57,051 DEBUG [Listener at localhost/43805-EventThread] zookeeper.ZKWatcher(600): regionserver:43507-0x10040e53a700003, quorum=127.0.0.1:55753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-02-11 01:00:57,051 DEBUG [Listener at localhost/43805-EventThread] zookeeper.ZKWatcher(600): regionserver:38923-0x10040e53a700002, quorum=127.0.0.1:55753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-02-11 01:00:57,051 DEBUG [Listener at localhost/43805-EventThread] zookeeper.ZKWatcher(600): regionserver:36543-0x10040e53a700001, quorum=127.0.0.1:55753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-02-11 01:00:57,051 DEBUG [Listener at localhost/43805-EventThread] zookeeper.ZKWatcher(600): master:46619-0x10040e53a700000, quorum=127.0.0.1:55753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-02-11 01:00:57,052 DEBUG [Listener at localhost/43805-EventThread] zookeeper.ZKWatcher(600): master:46619-0x10040e53a700000, quorum=127.0.0.1:55753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-11 01:00:57,053 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:46619-0x10040e53a700000, quorum=127.0.0.1:55753, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-02-11 01:00:57,054 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(224): Deleting ZNode for /hbase/backup-masters/jenkins-hbase4.apache.org,46619,1676077256908 from backup master directory 2023-02-11 01:00:57,054 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:46619-0x10040e53a700000, quorum=127.0.0.1:55753, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-02-11 01:00:57,055 DEBUG [Listener at localhost/43805-EventThread] zookeeper.ZKWatcher(600): master:46619-0x10040e53a700000, quorum=127.0.0.1:55753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase4.apache.org,46619,1676077256908 2023-02-11 01:00:57,056 DEBUG [Listener at localhost/43805-EventThread] zookeeper.ZKWatcher(600): master:46619-0x10040e53a700000, quorum=127.0.0.1:55753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-02-11 01:00:57,056 WARN [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-02-11 01:00:57,056 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(234): Registered as active master=jenkins-hbase4.apache.org,46619,1676077256908 2023-02-11 01:00:57,076 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] util.FSUtils(628): Created cluster ID file at hdfs://localhost:45625/user/jenkins/test-data/2a140438-d2b1-6994-16d5-13e6c7986734/hbase.id with ID: 1b84abeb-ab5a-4a07-9d47-c114b8f1704d 2023-02-11 01:00:57,092 INFO [master/jenkins-hbase4:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-11 01:00:57,096 DEBUG [Listener at localhost/43805-EventThread] zookeeper.ZKWatcher(600): master:46619-0x10040e53a700000, quorum=127.0.0.1:55753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-11 01:00:57,111 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x68d75474 to 127.0.0.1:55753 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-11 01:00:57,114 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3f5a7b15, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-11 01:00:57,115 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-02-11 01:00:57,115 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-02-11 01:00:57,115 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-11 01:00:57,117 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7689): Creating {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, under table dir hdfs://localhost:45625/user/jenkins/test-data/2a140438-d2b1-6994-16d5-13e6c7986734/MasterData/data/master/store-tmp 2023-02-11 01:00:57,130 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(865): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 01:00:57,130 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1603): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-02-11 01:00:57,130 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1625): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-11 01:00:57,130 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1646): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-11 01:00:57,130 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1713): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-02-11 01:00:57,130 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1723): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-11 01:00:57,130 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1837): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-11 01:00:57,130 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1557): Region close journal for 1595e783b53d99cd5eef43b6debb2682: Waiting for close lock at 1676077257130Disabling compacts and flushes for region at 1676077257130Disabling writes for close at 1676077257130Writing region close event to WAL at 1676077257130Closed at 1676077257130 2023-02-11 01:00:57,131 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(191): WALDir=hdfs://localhost:45625/user/jenkins/test-data/2a140438-d2b1-6994-16d5-13e6c7986734/MasterData/WALs/jenkins-hbase4.apache.org,46619,1676077256908 2023-02-11 01:00:57,135 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C46619%2C1676077256908, suffix=, logDir=hdfs://localhost:45625/user/jenkins/test-data/2a140438-d2b1-6994-16d5-13e6c7986734/MasterData/WALs/jenkins-hbase4.apache.org,46619,1676077256908, archiveDir=hdfs://localhost:45625/user/jenkins/test-data/2a140438-d2b1-6994-16d5-13e6c7986734/MasterData/oldWALs, maxLogs=10 2023-02-11 01:00:57,151 DEBUG [RS-EventLoopGroup-10-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:38851,DS-30862ee3-36ca-4a21-884e-547ca76a2efc,DISK] 2023-02-11 01:00:57,153 DEBUG [RS-EventLoopGroup-10-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:35587,DS-2350da3f-1592-4ca5-a29c-11cf6e912f79,DISK] 2023-02-11 01:00:57,153 DEBUG [RS-EventLoopGroup-10-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:43101,DS-2d423c52-201c-4032-9b49-b39dee42b6b7,DISK] 2023-02-11 01:00:57,156 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/2a140438-d2b1-6994-16d5-13e6c7986734/MasterData/WALs/jenkins-hbase4.apache.org,46619,1676077256908/jenkins-hbase4.apache.org%2C46619%2C1676077256908.1676077257135 2023-02-11 01:00:57,156 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:38851,DS-30862ee3-36ca-4a21-884e-547ca76a2efc,DISK], DatanodeInfoWithStorage[127.0.0.1:35587,DS-2350da3f-1592-4ca5-a29c-11cf6e912f79,DISK], DatanodeInfoWithStorage[127.0.0.1:43101,DS-2d423c52-201c-4032-9b49-b39dee42b6b7,DISK]] 2023-02-11 01:00:57,156 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7850): Opening region: {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''} 2023-02-11 01:00:57,157 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(865): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 01:00:57,157 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7890): checking encryption for 1595e783b53d99cd5eef43b6debb2682 2023-02-11 01:00:57,157 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7893): checking classloading for 1595e783b53d99cd5eef43b6debb2682 2023-02-11 01:00:57,161 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family proc of region 1595e783b53d99cd5eef43b6debb2682 2023-02-11 01:00:57,163 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45625/user/jenkins/test-data/2a140438-d2b1-6994-16d5-13e6c7986734/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc 2023-02-11 01:00:57,163 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc 2023-02-11 01:00:57,164 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(310): Store=1595e783b53d99cd5eef43b6debb2682/proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 01:00:57,165 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:45625/user/jenkins/test-data/2a140438-d2b1-6994-16d5-13e6c7986734/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-02-11 01:00:57,166 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:45625/user/jenkins/test-data/2a140438-d2b1-6994-16d5-13e6c7986734/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-02-11 01:00:57,169 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1054): writing seq id for 1595e783b53d99cd5eef43b6debb2682 2023-02-11 01:00:57,172 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45625/user/jenkins/test-data/2a140438-d2b1-6994-16d5-13e6c7986734/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-11 01:00:57,173 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1071): Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=67586327, jitterRate=0.0071147531270980835}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 01:00:57,173 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(964): Region open journal for 1595e783b53d99cd5eef43b6debb2682: Writing region info on filesystem at 1676077257157Initializing all the Stores at 1676077257158 (+1 ms)Instantiating store for column family {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676077257158Cleaning up temporary data from old regions at 1676077257166 (+8 ms)Cleaning up detritus from prior splits at 1676077257167 (+1 ms)Region opened successfully at 1676077257173 (+6 ms) 2023-02-11 01:00:57,173 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2023-02-11 01:00:57,174 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=5 2023-02-11 01:00:57,174 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(562): Starting 5 core workers (bigger of cpus/4 or 16) with max (burst) worker count=50 2023-02-11 01:00:57,174 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-02-11 01:00:57,175 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-02-11 01:00:57,175 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 0 msec 2023-02-11 01:00:57,175 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(95): Instantiated, coreThreads=3 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2023-02-11 01:00:57,176 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(253): hbase:meta replica znodes: [] 2023-02-11 01:00:57,177 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 0 possibly 'live' servers, and 0 'splitting'. 2023-02-11 01:00:57,188 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-02-11 01:00:57,188 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, ServerLocalityCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-02-11 01:00:57,188 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:46619-0x10040e53a700000, quorum=127.0.0.1:55753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2023-02-11 01:00:57,189 INFO [master/jenkins-hbase4:0:becomeActiveMaster] normalizer.RegionNormalizerWorker(118): Normalizer rate limit set to unlimited 2023-02-11 01:00:57,189 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:46619-0x10040e53a700000, quorum=127.0.0.1:55753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2023-02-11 01:00:57,191 DEBUG [Listener at localhost/43805-EventThread] zookeeper.ZKWatcher(600): master:46619-0x10040e53a700000, quorum=127.0.0.1:55753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-11 01:00:57,192 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:46619-0x10040e53a700000, quorum=127.0.0.1:55753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2023-02-11 01:00:57,193 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:46619-0x10040e53a700000, quorum=127.0.0.1:55753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2023-02-11 01:00:57,194 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:46619-0x10040e53a700000, quorum=127.0.0.1:55753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/snapshot-cleanup 2023-02-11 01:00:57,195 DEBUG [Listener at localhost/43805-EventThread] zookeeper.ZKWatcher(600): regionserver:38923-0x10040e53a700002, quorum=127.0.0.1:55753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-02-11 01:00:57,195 DEBUG [Listener at localhost/43805-EventThread] zookeeper.ZKWatcher(600): master:46619-0x10040e53a700000, quorum=127.0.0.1:55753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-02-11 01:00:57,195 DEBUG [Listener at localhost/43805-EventThread] zookeeper.ZKWatcher(600): regionserver:36543-0x10040e53a700001, quorum=127.0.0.1:55753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-02-11 01:00:57,195 DEBUG [Listener at localhost/43805-EventThread] zookeeper.ZKWatcher(600): master:46619-0x10040e53a700000, quorum=127.0.0.1:55753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-11 01:00:57,195 DEBUG [Listener at localhost/43805-EventThread] zookeeper.ZKWatcher(600): regionserver:43507-0x10040e53a700003, quorum=127.0.0.1:55753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-02-11 01:00:57,197 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(739): Active/primary master=jenkins-hbase4.apache.org,46619,1676077256908, sessionid=0x10040e53a700000, setting cluster-up flag (Was=false) 2023-02-11 01:00:57,201 DEBUG [Listener at localhost/43805-EventThread] zookeeper.ZKWatcher(600): master:46619-0x10040e53a700000, quorum=127.0.0.1:55753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-11 01:00:57,207 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2023-02-11 01:00:57,208 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,46619,1676077256908 2023-02-11 01:00:57,210 DEBUG [Listener at localhost/43805-EventThread] zookeeper.ZKWatcher(600): master:46619-0x10040e53a700000, quorum=127.0.0.1:55753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-11 01:00:57,216 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2023-02-11 01:00:57,217 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,46619,1676077256908 2023-02-11 01:00:57,218 WARN [master/jenkins-hbase4:0:becomeActiveMaster] snapshot.SnapshotManager(302): Couldn't delete working snapshot directory: hdfs://localhost:45625/user/jenkins/test-data/2a140438-d2b1-6994-16d5-13e6c7986734/.hbase-snapshot/.tmp 2023-02-11 01:00:57,220 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT; InitMetaProcedure table=hbase:meta 2023-02-11 01:00:57,221 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-02-11 01:00:57,221 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-02-11 01:00:57,221 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-02-11 01:00:57,221 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-02-11 01:00:57,221 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-master/jenkins-hbase4:0, corePoolSize=10, maxPoolSize=10 2023-02-11 01:00:57,221 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 01:00:57,221 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_MERGE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-02-11 01:00:57,221 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 01:00:57,229 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1676077287229 2023-02-11 01:00:57,229 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2023-02-11 01:00:57,229 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2023-02-11 01:00:57,229 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2023-02-11 01:00:57,230 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT, locked=true; InitMetaProcedure table=hbase:meta 2023-02-11 01:00:57,230 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2023-02-11 01:00:57,230 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2023-02-11 01:00:57,230 INFO [PEWorker-1] procedure.InitMetaProcedure(71): BOOTSTRAP: creating hbase:meta region 2023-02-11 01:00:57,230 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2023-02-11 01:00:57,230 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-02-11 01:00:57,231 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2023-02-11 01:00:57,231 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2023-02-11 01:00:57,231 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2023-02-11 01:00:57,231 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2023-02-11 01:00:57,231 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2023-02-11 01:00:57,231 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(242): Starting for large file=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1676077257231,5,FailOnTimeoutGroup] 2023-02-11 01:00:57,232 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(257): Starting for small files=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1676077257232,5,FailOnTimeoutGroup] 2023-02-11 01:00:57,232 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-02-11 01:00:57,232 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1451): Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2023-02-11 01:00:57,232 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2023-02-11 01:00:57,232 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2023-02-11 01:00:57,232 INFO [PEWorker-1] util.FSTableDescriptors(128): Creating new hbase:meta table descriptor 'hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-02-11 01:00:57,249 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:45625/user/jenkins/test-data/2a140438-d2b1-6994-16d5-13e6c7986734/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-02-11 01:00:57,249 INFO [RS:2;jenkins-hbase4:43507] regionserver.HRegionServer(952): ClusterId : 1b84abeb-ab5a-4a07-9d47-c114b8f1704d 2023-02-11 01:00:57,249 INFO [RS:1;jenkins-hbase4:38923] regionserver.HRegionServer(952): ClusterId : 1b84abeb-ab5a-4a07-9d47-c114b8f1704d 2023-02-11 01:00:57,249 INFO [RS:0;jenkins-hbase4:36543] regionserver.HRegionServer(952): ClusterId : 1b84abeb-ab5a-4a07-9d47-c114b8f1704d 2023-02-11 01:00:57,251 DEBUG [RS:1;jenkins-hbase4:38923] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-02-11 01:00:57,250 DEBUG [RS:2;jenkins-hbase4:43507] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-02-11 01:00:57,252 DEBUG [RS:0;jenkins-hbase4:36543] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-02-11 01:00:57,253 INFO [PEWorker-1] util.FSTableDescriptors(135): Updated hbase:meta table descriptor to hdfs://localhost:45625/user/jenkins/test-data/2a140438-d2b1-6994-16d5-13e6c7986734/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-02-11 01:00:57,253 INFO [PEWorker-1] regionserver.HRegion(7671): creating {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:45625/user/jenkins/test-data/2a140438-d2b1-6994-16d5-13e6c7986734 2023-02-11 01:00:57,255 DEBUG [RS:1;jenkins-hbase4:38923] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-02-11 01:00:57,255 DEBUG [RS:1;jenkins-hbase4:38923] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-02-11 01:00:57,255 DEBUG [RS:0;jenkins-hbase4:36543] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-02-11 01:00:57,256 DEBUG [RS:2;jenkins-hbase4:43507] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-02-11 01:00:57,261 DEBUG [RS:2;jenkins-hbase4:43507] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-02-11 01:00:57,260 DEBUG [RS:0;jenkins-hbase4:36543] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-02-11 01:00:57,263 DEBUG [RS:1;jenkins-hbase4:38923] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-02-11 01:00:57,264 DEBUG [RS:0;jenkins-hbase4:36543] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-02-11 01:00:57,264 DEBUG [RS:2;jenkins-hbase4:43507] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-02-11 01:00:57,270 DEBUG [RS:2;jenkins-hbase4:43507] zookeeper.ReadOnlyZKClient(139): Connect 0x0fffc065 to 127.0.0.1:55753 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-11 01:00:57,270 DEBUG [RS:0;jenkins-hbase4:36543] zookeeper.ReadOnlyZKClient(139): Connect 0x6a511774 to 127.0.0.1:55753 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-11 01:00:57,270 DEBUG [RS:1;jenkins-hbase4:38923] zookeeper.ReadOnlyZKClient(139): Connect 0x2cd7531e to 127.0.0.1:55753 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-11 01:00:57,277 DEBUG [RS:0;jenkins-hbase4:36543] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6e6e9ee6, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-11 01:00:57,277 DEBUG [RS:2;jenkins-hbase4:43507] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@bc04542, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-11 01:00:57,277 DEBUG [RS:0;jenkins-hbase4:36543] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2b6c1d4f, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-02-11 01:00:57,278 DEBUG [RS:2;jenkins-hbase4:43507] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4ec2ca5e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-02-11 01:00:57,278 DEBUG [RS:1;jenkins-hbase4:38923] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6dea5925, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-11 01:00:57,279 DEBUG [RS:1;jenkins-hbase4:38923] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4a14546d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-02-11 01:00:57,283 DEBUG [PEWorker-1] regionserver.HRegion(865): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 01:00:57,285 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-02-11 01:00:57,287 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45625/user/jenkins/test-data/2a140438-d2b1-6994-16d5-13e6c7986734/data/hbase/meta/1588230740/info 2023-02-11 01:00:57,287 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-02-11 01:00:57,288 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 01:00:57,288 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-02-11 01:00:57,289 DEBUG [RS:2;jenkins-hbase4:43507] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:2;jenkins-hbase4:43507 2023-02-11 01:00:57,289 DEBUG [RS:0;jenkins-hbase4:36543] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:0;jenkins-hbase4:36543 2023-02-11 01:00:57,289 INFO [RS:2;jenkins-hbase4:43507] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-02-11 01:00:57,290 INFO [RS:0;jenkins-hbase4:36543] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-02-11 01:00:57,290 INFO [RS:0;jenkins-hbase4:36543] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-02-11 01:00:57,290 INFO [RS:2;jenkins-hbase4:43507] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-02-11 01:00:57,290 DEBUG [RS:0;jenkins-hbase4:36543] regionserver.HRegionServer(1023): About to register with Master. 2023-02-11 01:00:57,290 DEBUG [RS:2;jenkins-hbase4:43507] regionserver.HRegionServer(1023): About to register with Master. 2023-02-11 01:00:57,290 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45625/user/jenkins/test-data/2a140438-d2b1-6994-16d5-13e6c7986734/data/hbase/meta/1588230740/rep_barrier 2023-02-11 01:00:57,290 INFO [RS:0;jenkins-hbase4:36543] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,46619,1676077256908 with isa=jenkins-hbase4.apache.org/172.31.14.131:36543, startcode=1676077256955 2023-02-11 01:00:57,290 INFO [RS:2;jenkins-hbase4:43507] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,46619,1676077256908 with isa=jenkins-hbase4.apache.org/172.31.14.131:43507, startcode=1676077257019 2023-02-11 01:00:57,291 DEBUG [RS:2;jenkins-hbase4:43507] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-02-11 01:00:57,291 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-02-11 01:00:57,291 DEBUG [RS:0;jenkins-hbase4:36543] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-02-11 01:00:57,292 DEBUG [RS:1;jenkins-hbase4:38923] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:1;jenkins-hbase4:38923 2023-02-11 01:00:57,293 INFO [RS:1;jenkins-hbase4:38923] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-02-11 01:00:57,293 INFO [RS:1;jenkins-hbase4:38923] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-02-11 01:00:57,293 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 01:00:57,293 DEBUG [RS:1;jenkins-hbase4:38923] regionserver.HRegionServer(1023): About to register with Master. 2023-02-11 01:00:57,293 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-02-11 01:00:57,295 INFO [RS:1;jenkins-hbase4:38923] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,46619,1676077256908 with isa=jenkins-hbase4.apache.org/172.31.14.131:38923, startcode=1676077256985 2023-02-11 01:00:57,296 DEBUG [RS:1;jenkins-hbase4:38923] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-02-11 01:00:57,296 INFO [RS-EventLoopGroup-7-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:47685, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins.hfs.5 (auth:SIMPLE), service=RegionServerStatusService 2023-02-11 01:00:57,297 INFO [RS-EventLoopGroup-7-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46391, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins.hfs.3 (auth:SIMPLE), service=RegionServerStatusService 2023-02-11 01:00:57,298 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46619] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,43507,1676077257019 2023-02-11 01:00:57,298 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46619] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,36543,1676077256955 2023-02-11 01:00:57,299 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45625/user/jenkins/test-data/2a140438-d2b1-6994-16d5-13e6c7986734/data/hbase/meta/1588230740/table 2023-02-11 01:00:57,299 INFO [RS-EventLoopGroup-7-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39617, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins.hfs.4 (auth:SIMPLE), service=RegionServerStatusService 2023-02-11 01:00:57,300 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=46619] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,38923,1676077256985 2023-02-11 01:00:57,300 DEBUG [RS:2;jenkins-hbase4:43507] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost:45625/user/jenkins/test-data/2a140438-d2b1-6994-16d5-13e6c7986734 2023-02-11 01:00:57,300 DEBUG [RS:2;jenkins-hbase4:43507] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost:45625 2023-02-11 01:00:57,300 DEBUG [RS:2;jenkins-hbase4:43507] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-02-11 01:00:57,300 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-02-11 01:00:57,300 DEBUG [RS:1;jenkins-hbase4:38923] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost:45625/user/jenkins/test-data/2a140438-d2b1-6994-16d5-13e6c7986734 2023-02-11 01:00:57,300 DEBUG [RS:1;jenkins-hbase4:38923] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost:45625 2023-02-11 01:00:57,300 DEBUG [RS:1;jenkins-hbase4:38923] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-02-11 01:00:57,301 DEBUG [RS:0;jenkins-hbase4:36543] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost:45625/user/jenkins/test-data/2a140438-d2b1-6994-16d5-13e6c7986734 2023-02-11 01:00:57,301 DEBUG [RS:0;jenkins-hbase4:36543] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost:45625 2023-02-11 01:00:57,301 DEBUG [RS:0;jenkins-hbase4:36543] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-02-11 01:00:57,303 DEBUG [Listener at localhost/43805-EventThread] zookeeper.ZKWatcher(600): master:46619-0x10040e53a700000, quorum=127.0.0.1:55753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-11 01:00:57,304 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 01:00:57,305 DEBUG [PEWorker-1] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:45625/user/jenkins/test-data/2a140438-d2b1-6994-16d5-13e6c7986734/data/hbase/meta/1588230740 2023-02-11 01:00:57,306 DEBUG [PEWorker-1] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:45625/user/jenkins/test-data/2a140438-d2b1-6994-16d5-13e6c7986734/data/hbase/meta/1588230740 2023-02-11 01:00:57,307 DEBUG [RS:2;jenkins-hbase4:43507] zookeeper.ZKUtil(162): regionserver:43507-0x10040e53a700003, quorum=127.0.0.1:55753, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,43507,1676077257019 2023-02-11 01:00:57,307 WARN [RS:2;jenkins-hbase4:43507] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-02-11 01:00:57,307 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,43507,1676077257019] 2023-02-11 01:00:57,307 INFO [RS:2;jenkins-hbase4:43507] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-11 01:00:57,307 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,36543,1676077256955] 2023-02-11 01:00:57,307 DEBUG [RS:2;jenkins-hbase4:43507] regionserver.HRegionServer(1947): logDir=hdfs://localhost:45625/user/jenkins/test-data/2a140438-d2b1-6994-16d5-13e6c7986734/WALs/jenkins-hbase4.apache.org,43507,1676077257019 2023-02-11 01:00:57,307 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,38923,1676077256985] 2023-02-11 01:00:57,310 DEBUG [PEWorker-1] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-02-11 01:00:57,312 DEBUG [PEWorker-1] regionserver.HRegion(1054): writing seq id for 1588230740 2023-02-11 01:00:57,313 DEBUG [RS:0;jenkins-hbase4:36543] zookeeper.ZKUtil(162): regionserver:36543-0x10040e53a700001, quorum=127.0.0.1:55753, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,36543,1676077256955 2023-02-11 01:00:57,313 DEBUG [RS:1;jenkins-hbase4:38923] zookeeper.ZKUtil(162): regionserver:38923-0x10040e53a700002, quorum=127.0.0.1:55753, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,38923,1676077256985 2023-02-11 01:00:57,313 WARN [RS:0;jenkins-hbase4:36543] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-02-11 01:00:57,313 WARN [RS:1;jenkins-hbase4:38923] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-02-11 01:00:57,313 INFO [RS:0;jenkins-hbase4:36543] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-11 01:00:57,313 INFO [RS:1;jenkins-hbase4:38923] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-11 01:00:57,313 DEBUG [RS:0;jenkins-hbase4:36543] regionserver.HRegionServer(1947): logDir=hdfs://localhost:45625/user/jenkins/test-data/2a140438-d2b1-6994-16d5-13e6c7986734/WALs/jenkins-hbase4.apache.org,36543,1676077256955 2023-02-11 01:00:57,314 DEBUG [RS:1;jenkins-hbase4:38923] regionserver.HRegionServer(1947): logDir=hdfs://localhost:45625/user/jenkins/test-data/2a140438-d2b1-6994-16d5-13e6c7986734/WALs/jenkins-hbase4.apache.org,38923,1676077256985 2023-02-11 01:00:57,316 DEBUG [RS:2;jenkins-hbase4:43507] zookeeper.ZKUtil(162): regionserver:43507-0x10040e53a700003, quorum=127.0.0.1:55753, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,43507,1676077257019 2023-02-11 01:00:57,316 DEBUG [RS:2;jenkins-hbase4:43507] zookeeper.ZKUtil(162): regionserver:43507-0x10040e53a700003, quorum=127.0.0.1:55753, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,36543,1676077256955 2023-02-11 01:00:57,317 DEBUG [RS:2;jenkins-hbase4:43507] zookeeper.ZKUtil(162): regionserver:43507-0x10040e53a700003, quorum=127.0.0.1:55753, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,38923,1676077256985 2023-02-11 01:00:57,318 DEBUG [RS:2;jenkins-hbase4:43507] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-02-11 01:00:57,318 INFO [RS:2;jenkins-hbase4:43507] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-02-11 01:00:57,324 INFO [RS:2;jenkins-hbase4:43507] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-02-11 01:00:57,328 INFO [RS:2;jenkins-hbase4:43507] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-02-11 01:00:57,328 INFO [RS:2;jenkins-hbase4:43507] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-02-11 01:00:57,329 INFO [RS:2;jenkins-hbase4:43507] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-02-11 01:00:57,329 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45625/user/jenkins/test-data/2a140438-d2b1-6994-16d5-13e6c7986734/data/hbase/meta/1588230740/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-11 01:00:57,331 INFO [PEWorker-1] regionserver.HRegion(1071): Opened 1588230740; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=68218636, jitterRate=0.0165368914604187}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-02-11 01:00:57,331 DEBUG [PEWorker-1] regionserver.HRegion(964): Region open journal for 1588230740: Writing region info on filesystem at 1676077257283Initializing all the Stores at 1676077257284 (+1 ms)Instantiating store for column family {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} at 1676077257284Instantiating store for column family {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676077257284Instantiating store for column family {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} at 1676077257284Cleaning up temporary data from old regions at 1676077257308 (+24 ms)Cleaning up detritus from prior splits at 1676077257309 (+1 ms)Region opened successfully at 1676077257331 (+22 ms) 2023-02-11 01:00:57,332 DEBUG [PEWorker-1] regionserver.HRegion(1603): Closing 1588230740, disabling compactions & flushes 2023-02-11 01:00:57,332 INFO [PEWorker-1] regionserver.HRegion(1625): Closing region hbase:meta,,1.1588230740 2023-02-11 01:00:57,332 DEBUG [PEWorker-1] regionserver.HRegion(1646): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-02-11 01:00:57,332 DEBUG [PEWorker-1] regionserver.HRegion(1713): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-02-11 01:00:57,332 DEBUG [PEWorker-1] regionserver.HRegion(1723): Updates disabled for region hbase:meta,,1.1588230740 2023-02-11 01:00:57,332 DEBUG [RS:0;jenkins-hbase4:36543] zookeeper.ZKUtil(162): regionserver:36543-0x10040e53a700001, quorum=127.0.0.1:55753, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,43507,1676077257019 2023-02-11 01:00:57,336 INFO [RS:2;jenkins-hbase4:43507] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-02-11 01:00:57,332 DEBUG [RS:1;jenkins-hbase4:38923] zookeeper.ZKUtil(162): regionserver:38923-0x10040e53a700002, quorum=127.0.0.1:55753, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,43507,1676077257019 2023-02-11 01:00:57,336 DEBUG [RS:2;jenkins-hbase4:43507] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 01:00:57,336 INFO [PEWorker-1] regionserver.HRegion(1837): Closed hbase:meta,,1.1588230740 2023-02-11 01:00:57,337 DEBUG [RS:2;jenkins-hbase4:43507] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 01:00:57,337 DEBUG [RS:0;jenkins-hbase4:36543] zookeeper.ZKUtil(162): regionserver:36543-0x10040e53a700001, quorum=127.0.0.1:55753, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,36543,1676077256955 2023-02-11 01:00:57,337 DEBUG [RS:1;jenkins-hbase4:38923] zookeeper.ZKUtil(162): regionserver:38923-0x10040e53a700002, quorum=127.0.0.1:55753, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,36543,1676077256955 2023-02-11 01:00:57,337 DEBUG [RS:2;jenkins-hbase4:43507] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 01:00:57,337 DEBUG [PEWorker-1] regionserver.HRegion(1557): Region close journal for 1588230740: Waiting for close lock at 1676077257332Disabling compacts and flushes for region at 1676077257332Disabling writes for close at 1676077257332Writing region close event to WAL at 1676077257336 (+4 ms)Closed at 1676077257336 2023-02-11 01:00:57,337 DEBUG [RS:2;jenkins-hbase4:43507] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 01:00:57,337 DEBUG [RS:2;jenkins-hbase4:43507] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 01:00:57,337 DEBUG [RS:2;jenkins-hbase4:43507] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-02-11 01:00:57,337 DEBUG [RS:1;jenkins-hbase4:38923] zookeeper.ZKUtil(162): regionserver:38923-0x10040e53a700002, quorum=127.0.0.1:55753, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,38923,1676077256985 2023-02-11 01:00:57,337 DEBUG [RS:2;jenkins-hbase4:43507] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 01:00:57,337 DEBUG [RS:0;jenkins-hbase4:36543] zookeeper.ZKUtil(162): regionserver:36543-0x10040e53a700001, quorum=127.0.0.1:55753, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,38923,1676077256985 2023-02-11 01:00:57,337 DEBUG [RS:2;jenkins-hbase4:43507] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 01:00:57,338 DEBUG [RS:2;jenkins-hbase4:43507] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 01:00:57,338 DEBUG [RS:2;jenkins-hbase4:43507] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 01:00:57,339 DEBUG [RS:1;jenkins-hbase4:38923] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-02-11 01:00:57,339 INFO [RS:1;jenkins-hbase4:38923] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-02-11 01:00:57,339 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_ASSIGN_META, locked=true; InitMetaProcedure table=hbase:meta 2023-02-11 01:00:57,339 INFO [PEWorker-1] procedure.InitMetaProcedure(103): Going to assign meta 2023-02-11 01:00:57,340 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-02-11 01:00:57,341 DEBUG [RS:0;jenkins-hbase4:36543] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-02-11 01:00:57,342 INFO [RS:0;jenkins-hbase4:36543] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-02-11 01:00:57,348 INFO [RS:2;jenkins-hbase4:43507] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-02-11 01:00:57,348 INFO [RS:2;jenkins-hbase4:43507] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-02-11 01:00:57,348 INFO [RS:2;jenkins-hbase4:43507] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-02-11 01:00:57,348 INFO [RS:1;jenkins-hbase4:38923] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-02-11 01:00:57,349 INFO [RS:0;jenkins-hbase4:36543] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-02-11 01:00:57,350 INFO [RS:1;jenkins-hbase4:38923] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-02-11 01:00:57,352 INFO [RS:1;jenkins-hbase4:38923] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-02-11 01:00:57,350 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-02-11 01:00:57,354 INFO [RS:0;jenkins-hbase4:36543] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-02-11 01:00:57,354 INFO [RS:0;jenkins-hbase4:36543] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-02-11 01:00:57,354 INFO [RS:1;jenkins-hbase4:38923] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-02-11 01:00:57,355 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OFFLINE, location=null; forceNewPlan=false, retain=false 2023-02-11 01:00:57,360 INFO [RS:0;jenkins-hbase4:36543] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-02-11 01:00:57,360 INFO [RS:1;jenkins-hbase4:38923] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-02-11 01:00:57,360 DEBUG [RS:1;jenkins-hbase4:38923] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 01:00:57,360 DEBUG [RS:1;jenkins-hbase4:38923] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 01:00:57,361 DEBUG [RS:1;jenkins-hbase4:38923] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 01:00:57,361 DEBUG [RS:1;jenkins-hbase4:38923] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 01:00:57,361 DEBUG [RS:1;jenkins-hbase4:38923] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 01:00:57,361 DEBUG [RS:1;jenkins-hbase4:38923] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-02-11 01:00:57,361 DEBUG [RS:1;jenkins-hbase4:38923] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 01:00:57,362 DEBUG [RS:1;jenkins-hbase4:38923] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 01:00:57,362 DEBUG [RS:1;jenkins-hbase4:38923] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 01:00:57,362 DEBUG [RS:1;jenkins-hbase4:38923] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 01:00:57,363 INFO [RS:1;jenkins-hbase4:38923] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-02-11 01:00:57,363 INFO [RS:0;jenkins-hbase4:36543] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-02-11 01:00:57,363 INFO [RS:1;jenkins-hbase4:38923] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-02-11 01:00:57,363 INFO [RS:1;jenkins-hbase4:38923] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-02-11 01:00:57,363 DEBUG [RS:0;jenkins-hbase4:36543] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 01:00:57,364 DEBUG [RS:0;jenkins-hbase4:36543] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 01:00:57,364 DEBUG [RS:0;jenkins-hbase4:36543] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 01:00:57,364 DEBUG [RS:0;jenkins-hbase4:36543] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 01:00:57,364 DEBUG [RS:0;jenkins-hbase4:36543] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 01:00:57,364 DEBUG [RS:0;jenkins-hbase4:36543] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-02-11 01:00:57,364 DEBUG [RS:0;jenkins-hbase4:36543] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 01:00:57,364 DEBUG [RS:0;jenkins-hbase4:36543] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 01:00:57,364 DEBUG [RS:0;jenkins-hbase4:36543] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 01:00:57,364 DEBUG [RS:0;jenkins-hbase4:36543] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-11 01:00:57,367 INFO [RS:2;jenkins-hbase4:43507] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-02-11 01:00:57,367 INFO [RS:2;jenkins-hbase4:43507] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,43507,1676077257019-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-02-11 01:00:57,370 INFO [RS:0;jenkins-hbase4:36543] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-02-11 01:00:57,371 INFO [RS:0;jenkins-hbase4:36543] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-02-11 01:00:57,371 INFO [RS:0;jenkins-hbase4:36543] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-02-11 01:00:57,378 INFO [RS:1;jenkins-hbase4:38923] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-02-11 01:00:57,378 INFO [RS:1;jenkins-hbase4:38923] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,38923,1676077256985-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-02-11 01:00:57,383 INFO [RS:2;jenkins-hbase4:43507] regionserver.Replication(203): jenkins-hbase4.apache.org,43507,1676077257019 started 2023-02-11 01:00:57,383 INFO [RS:2;jenkins-hbase4:43507] regionserver.HRegionServer(1638): Serving as jenkins-hbase4.apache.org,43507,1676077257019, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:43507, sessionid=0x10040e53a700003 2023-02-11 01:00:57,383 DEBUG [RS:2;jenkins-hbase4:43507] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-02-11 01:00:57,383 DEBUG [RS:2;jenkins-hbase4:43507] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,43507,1676077257019 2023-02-11 01:00:57,383 DEBUG [RS:2;jenkins-hbase4:43507] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,43507,1676077257019' 2023-02-11 01:00:57,383 DEBUG [RS:2;jenkins-hbase4:43507] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-11 01:00:57,384 DEBUG [RS:2;jenkins-hbase4:43507] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-11 01:00:57,384 DEBUG [RS:2;jenkins-hbase4:43507] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-02-11 01:00:57,384 DEBUG [RS:2;jenkins-hbase4:43507] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-02-11 01:00:57,385 DEBUG [RS:2;jenkins-hbase4:43507] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,43507,1676077257019 2023-02-11 01:00:57,385 DEBUG [RS:2;jenkins-hbase4:43507] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,43507,1676077257019' 2023-02-11 01:00:57,385 DEBUG [RS:2;jenkins-hbase4:43507] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-02-11 01:00:57,385 DEBUG [RS:2;jenkins-hbase4:43507] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-02-11 01:00:57,385 DEBUG [RS:2;jenkins-hbase4:43507] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-02-11 01:00:57,385 INFO [RS:2;jenkins-hbase4:43507] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-02-11 01:00:57,385 INFO [RS:2;jenkins-hbase4:43507] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-02-11 01:00:57,390 INFO [RS:0;jenkins-hbase4:36543] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-02-11 01:00:57,390 INFO [RS:0;jenkins-hbase4:36543] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,36543,1676077256955-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-02-11 01:00:57,391 INFO [RS:1;jenkins-hbase4:38923] regionserver.Replication(203): jenkins-hbase4.apache.org,38923,1676077256985 started 2023-02-11 01:00:57,391 INFO [RS:1;jenkins-hbase4:38923] regionserver.HRegionServer(1638): Serving as jenkins-hbase4.apache.org,38923,1676077256985, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:38923, sessionid=0x10040e53a700002 2023-02-11 01:00:57,391 DEBUG [RS:1;jenkins-hbase4:38923] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-02-11 01:00:57,391 DEBUG [RS:1;jenkins-hbase4:38923] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,38923,1676077256985 2023-02-11 01:00:57,392 DEBUG [RS:1;jenkins-hbase4:38923] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,38923,1676077256985' 2023-02-11 01:00:57,392 DEBUG [RS:1;jenkins-hbase4:38923] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-11 01:00:57,392 DEBUG [RS:1;jenkins-hbase4:38923] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-11 01:00:57,392 DEBUG [RS:1;jenkins-hbase4:38923] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-02-11 01:00:57,392 DEBUG [RS:1;jenkins-hbase4:38923] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-02-11 01:00:57,392 DEBUG [RS:1;jenkins-hbase4:38923] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,38923,1676077256985 2023-02-11 01:00:57,393 DEBUG [RS:1;jenkins-hbase4:38923] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,38923,1676077256985' 2023-02-11 01:00:57,393 DEBUG [RS:1;jenkins-hbase4:38923] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-02-11 01:00:57,393 DEBUG [RS:1;jenkins-hbase4:38923] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-02-11 01:00:57,393 DEBUG [RS:1;jenkins-hbase4:38923] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-02-11 01:00:57,393 INFO [RS:1;jenkins-hbase4:38923] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-02-11 01:00:57,393 INFO [RS:1;jenkins-hbase4:38923] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-02-11 01:00:57,406 INFO [RS:0;jenkins-hbase4:36543] regionserver.Replication(203): jenkins-hbase4.apache.org,36543,1676077256955 started 2023-02-11 01:00:57,406 INFO [RS:0;jenkins-hbase4:36543] regionserver.HRegionServer(1638): Serving as jenkins-hbase4.apache.org,36543,1676077256955, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:36543, sessionid=0x10040e53a700001 2023-02-11 01:00:57,406 DEBUG [RS:0;jenkins-hbase4:36543] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-02-11 01:00:57,406 DEBUG [RS:0;jenkins-hbase4:36543] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,36543,1676077256955 2023-02-11 01:00:57,406 DEBUG [RS:0;jenkins-hbase4:36543] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,36543,1676077256955' 2023-02-11 01:00:57,406 DEBUG [RS:0;jenkins-hbase4:36543] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-11 01:00:57,407 DEBUG [RS:0;jenkins-hbase4:36543] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-11 01:00:57,407 DEBUG [RS:0;jenkins-hbase4:36543] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-02-11 01:00:57,407 DEBUG [RS:0;jenkins-hbase4:36543] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-02-11 01:00:57,407 DEBUG [RS:0;jenkins-hbase4:36543] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,36543,1676077256955 2023-02-11 01:00:57,407 DEBUG [RS:0;jenkins-hbase4:36543] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,36543,1676077256955' 2023-02-11 01:00:57,407 DEBUG [RS:0;jenkins-hbase4:36543] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-02-11 01:00:57,407 DEBUG [RS:0;jenkins-hbase4:36543] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-02-11 01:00:57,408 DEBUG [RS:0;jenkins-hbase4:36543] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-02-11 01:00:57,408 INFO [RS:0;jenkins-hbase4:36543] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-02-11 01:00:57,408 INFO [RS:0;jenkins-hbase4:36543] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-02-11 01:00:57,488 INFO [RS:2;jenkins-hbase4:43507] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C43507%2C1676077257019, suffix=, logDir=hdfs://localhost:45625/user/jenkins/test-data/2a140438-d2b1-6994-16d5-13e6c7986734/WALs/jenkins-hbase4.apache.org,43507,1676077257019, archiveDir=hdfs://localhost:45625/user/jenkins/test-data/2a140438-d2b1-6994-16d5-13e6c7986734/oldWALs, maxLogs=32 2023-02-11 01:00:57,496 INFO [RS:1;jenkins-hbase4:38923] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C38923%2C1676077256985, suffix=, logDir=hdfs://localhost:45625/user/jenkins/test-data/2a140438-d2b1-6994-16d5-13e6c7986734/WALs/jenkins-hbase4.apache.org,38923,1676077256985, archiveDir=hdfs://localhost:45625/user/jenkins/test-data/2a140438-d2b1-6994-16d5-13e6c7986734/oldWALs, maxLogs=32 2023-02-11 01:00:57,510 DEBUG [RS-EventLoopGroup-10-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:43101,DS-2d423c52-201c-4032-9b49-b39dee42b6b7,DISK] 2023-02-11 01:00:57,510 DEBUG [RS-EventLoopGroup-10-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:35587,DS-2350da3f-1592-4ca5-a29c-11cf6e912f79,DISK] 2023-02-11 01:00:57,510 DEBUG [RS-EventLoopGroup-10-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:38851,DS-30862ee3-36ca-4a21-884e-547ca76a2efc,DISK] 2023-02-11 01:00:57,510 DEBUG [jenkins-hbase4:46619] assignment.AssignmentManager(2178): Processing assignQueue; systemServersCount=3, allServersCount=3 2023-02-11 01:00:57,510 DEBUG [jenkins-hbase4:46619] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-02-11 01:00:57,513 DEBUG [jenkins-hbase4:46619] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-11 01:00:57,513 DEBUG [jenkins-hbase4:46619] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-11 01:00:57,513 DEBUG [jenkins-hbase4:46619] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-11 01:00:57,513 DEBUG [jenkins-hbase4:46619] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-11 01:00:57,513 INFO [RS:0;jenkins-hbase4:36543] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C36543%2C1676077256955, suffix=, logDir=hdfs://localhost:45625/user/jenkins/test-data/2a140438-d2b1-6994-16d5-13e6c7986734/WALs/jenkins-hbase4.apache.org,36543,1676077256955, archiveDir=hdfs://localhost:45625/user/jenkins/test-data/2a140438-d2b1-6994-16d5-13e6c7986734/oldWALs, maxLogs=32 2023-02-11 01:00:57,517 INFO [PEWorker-2] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,38923,1676077256985, state=OPENING 2023-02-11 01:00:57,521 INFO [RS:2;jenkins-hbase4:43507] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/2a140438-d2b1-6994-16d5-13e6c7986734/WALs/jenkins-hbase4.apache.org,43507,1676077257019/jenkins-hbase4.apache.org%2C43507%2C1676077257019.1676077257489 2023-02-11 01:00:57,521 DEBUG [RS:2;jenkins-hbase4:43507] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:38851,DS-30862ee3-36ca-4a21-884e-547ca76a2efc,DISK], DatanodeInfoWithStorage[127.0.0.1:43101,DS-2d423c52-201c-4032-9b49-b39dee42b6b7,DISK], DatanodeInfoWithStorage[127.0.0.1:35587,DS-2350da3f-1592-4ca5-a29c-11cf6e912f79,DISK]] 2023-02-11 01:00:57,521 DEBUG [PEWorker-2] zookeeper.MetaTableLocator(240): hbase:meta region location doesn't exist, create it 2023-02-11 01:00:57,525 DEBUG [Listener at localhost/43805-EventThread] zookeeper.ZKWatcher(600): master:46619-0x10040e53a700000, quorum=127.0.0.1:55753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-11 01:00:57,526 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-02-11 01:00:57,526 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=3, ppid=2, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,38923,1676077256985}] 2023-02-11 01:00:57,526 DEBUG [RS-EventLoopGroup-10-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:43101,DS-2d423c52-201c-4032-9b49-b39dee42b6b7,DISK] 2023-02-11 01:00:57,528 DEBUG [RS-EventLoopGroup-10-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:38851,DS-30862ee3-36ca-4a21-884e-547ca76a2efc,DISK] 2023-02-11 01:00:57,528 DEBUG [RS-EventLoopGroup-10-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:35587,DS-2350da3f-1592-4ca5-a29c-11cf6e912f79,DISK] 2023-02-11 01:00:57,533 INFO [RS:1;jenkins-hbase4:38923] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/2a140438-d2b1-6994-16d5-13e6c7986734/WALs/jenkins-hbase4.apache.org,38923,1676077256985/jenkins-hbase4.apache.org%2C38923%2C1676077256985.1676077257497 2023-02-11 01:00:57,533 DEBUG [RS:1;jenkins-hbase4:38923] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:43101,DS-2d423c52-201c-4032-9b49-b39dee42b6b7,DISK], DatanodeInfoWithStorage[127.0.0.1:35587,DS-2350da3f-1592-4ca5-a29c-11cf6e912f79,DISK], DatanodeInfoWithStorage[127.0.0.1:38851,DS-30862ee3-36ca-4a21-884e-547ca76a2efc,DISK]] 2023-02-11 01:00:57,539 DEBUG [RS-EventLoopGroup-10-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:35587,DS-2350da3f-1592-4ca5-a29c-11cf6e912f79,DISK] 2023-02-11 01:00:57,539 DEBUG [RS-EventLoopGroup-10-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:43101,DS-2d423c52-201c-4032-9b49-b39dee42b6b7,DISK] 2023-02-11 01:00:57,539 DEBUG [RS-EventLoopGroup-10-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:38851,DS-30862ee3-36ca-4a21-884e-547ca76a2efc,DISK] 2023-02-11 01:00:57,542 INFO [RS:0;jenkins-hbase4:36543] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/2a140438-d2b1-6994-16d5-13e6c7986734/WALs/jenkins-hbase4.apache.org,36543,1676077256955/jenkins-hbase4.apache.org%2C36543%2C1676077256955.1676077257514 2023-02-11 01:00:57,544 DEBUG [RS:0;jenkins-hbase4:36543] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:35587,DS-2350da3f-1592-4ca5-a29c-11cf6e912f79,DISK], DatanodeInfoWithStorage[127.0.0.1:43101,DS-2d423c52-201c-4032-9b49-b39dee42b6b7,DISK], DatanodeInfoWithStorage[127.0.0.1:38851,DS-30862ee3-36ca-4a21-884e-547ca76a2efc,DISK]] 2023-02-11 01:00:57,682 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,38923,1676077256985 2023-02-11 01:00:57,683 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-02-11 01:00:57,684 INFO [RS-EventLoopGroup-9-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39726, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-02-11 01:00:57,689 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open hbase:meta,,1.1588230740 2023-02-11 01:00:57,689 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-11 01:00:57,691 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C38923%2C1676077256985.meta, suffix=.meta, logDir=hdfs://localhost:45625/user/jenkins/test-data/2a140438-d2b1-6994-16d5-13e6c7986734/WALs/jenkins-hbase4.apache.org,38923,1676077256985, archiveDir=hdfs://localhost:45625/user/jenkins/test-data/2a140438-d2b1-6994-16d5-13e6c7986734/oldWALs, maxLogs=32 2023-02-11 01:00:57,706 DEBUG [RS-EventLoopGroup-10-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:38851,DS-30862ee3-36ca-4a21-884e-547ca76a2efc,DISK] 2023-02-11 01:00:57,706 DEBUG [RS-EventLoopGroup-10-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:35587,DS-2350da3f-1592-4ca5-a29c-11cf6e912f79,DISK] 2023-02-11 01:00:57,706 DEBUG [RS-EventLoopGroup-10-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:43101,DS-2d423c52-201c-4032-9b49-b39dee42b6b7,DISK] 2023-02-11 01:00:57,709 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/2a140438-d2b1-6994-16d5-13e6c7986734/WALs/jenkins-hbase4.apache.org,38923,1676077256985/jenkins-hbase4.apache.org%2C38923%2C1676077256985.meta.1676077257692.meta 2023-02-11 01:00:57,709 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:38851,DS-30862ee3-36ca-4a21-884e-547ca76a2efc,DISK], DatanodeInfoWithStorage[127.0.0.1:43101,DS-2d423c52-201c-4032-9b49-b39dee42b6b7,DISK], DatanodeInfoWithStorage[127.0.0.1:35587,DS-2350da3f-1592-4ca5-a29c-11cf6e912f79,DISK]] 2023-02-11 01:00:57,709 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-02-11 01:00:57,710 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-02-11 01:00:57,710 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(8546): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-02-11 01:00:57,712 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-02-11 01:00:57,712 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-02-11 01:00:57,712 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 01:00:57,712 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 1588230740 2023-02-11 01:00:57,712 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 1588230740 2023-02-11 01:00:57,714 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-02-11 01:00:57,715 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45625/user/jenkins/test-data/2a140438-d2b1-6994-16d5-13e6c7986734/data/hbase/meta/1588230740/info 2023-02-11 01:00:57,715 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45625/user/jenkins/test-data/2a140438-d2b1-6994-16d5-13e6c7986734/data/hbase/meta/1588230740/info 2023-02-11 01:00:57,716 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-02-11 01:00:57,717 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 01:00:57,717 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-02-11 01:00:57,718 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45625/user/jenkins/test-data/2a140438-d2b1-6994-16d5-13e6c7986734/data/hbase/meta/1588230740/rep_barrier 2023-02-11 01:00:57,718 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45625/user/jenkins/test-data/2a140438-d2b1-6994-16d5-13e6c7986734/data/hbase/meta/1588230740/rep_barrier 2023-02-11 01:00:57,718 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-02-11 01:00:57,719 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 01:00:57,719 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-02-11 01:00:57,720 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45625/user/jenkins/test-data/2a140438-d2b1-6994-16d5-13e6c7986734/data/hbase/meta/1588230740/table 2023-02-11 01:00:57,720 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45625/user/jenkins/test-data/2a140438-d2b1-6994-16d5-13e6c7986734/data/hbase/meta/1588230740/table 2023-02-11 01:00:57,720 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-02-11 01:00:57,721 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 01:00:57,722 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:45625/user/jenkins/test-data/2a140438-d2b1-6994-16d5-13e6c7986734/data/hbase/meta/1588230740 2023-02-11 01:00:57,723 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:45625/user/jenkins/test-data/2a140438-d2b1-6994-16d5-13e6c7986734/data/hbase/meta/1588230740 2023-02-11 01:00:57,727 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-02-11 01:00:57,728 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 1588230740 2023-02-11 01:00:57,729 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 1588230740; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=64973089, jitterRate=-0.03182552754878998}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-02-11 01:00:57,729 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 1588230740: Running coprocessor pre-open hook at 1676077257712Writing region info on filesystem at 1676077257712Initializing all the Stores at 1676077257713 (+1 ms)Instantiating store for column family {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} at 1676077257713Instantiating store for column family {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676077257714 (+1 ms)Instantiating store for column family {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} at 1676077257714Cleaning up temporary data from old regions at 1676077257724 (+10 ms)Cleaning up detritus from prior splits at 1676077257726 (+2 ms)Running coprocessor post-open hooks at 1676077257729 (+3 ms)Region opened successfully at 1676077257729 2023-02-11 01:00:57,731 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:meta,,1.1588230740, pid=3, masterSystemTime=1676077257682 2023-02-11 01:00:57,735 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:meta,,1.1588230740 2023-02-11 01:00:57,735 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened hbase:meta,,1.1588230740 2023-02-11 01:00:57,737 INFO [PEWorker-5] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,38923,1676077256985, state=OPEN 2023-02-11 01:00:57,739 DEBUG [Listener at localhost/43805-EventThread] zookeeper.ZKWatcher(600): master:46619-0x10040e53a700000, quorum=127.0.0.1:55753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-02-11 01:00:57,739 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-02-11 01:00:57,742 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=3, resume processing ppid=2 2023-02-11 01:00:57,742 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=3, ppid=2, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,38923,1676077256985 in 213 msec 2023-02-11 01:00:57,746 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=2, resume processing ppid=1 2023-02-11 01:00:57,746 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=2, ppid=1, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 403 msec 2023-02-11 01:00:57,749 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta in 529 msec 2023-02-11 01:00:57,749 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(946): Master startup: status=Wait for region servers to report in, state=RUNNING, startTime=1676077257050, completionTime=-1 2023-02-11 01:00:57,749 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ServerManager(821): Finished waiting on RegionServer count=3; waited=0ms, expected min=3 server(s), max=3 server(s), master is running 2023-02-11 01:00:57,749 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1519): Joining cluster... 2023-02-11 01:00:57,752 DEBUG [hconnection-0x76047910-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-11 01:00:57,753 INFO [RS-EventLoopGroup-9-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39740, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-11 01:00:57,755 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1531): Number of RegionServers=3 2023-02-11 01:00:57,755 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1676077317755 2023-02-11 01:00:57,755 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1676077377755 2023-02-11 01:00:57,755 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1538): Joined the cluster in 6 msec 2023-02-11 01:00:57,761 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,46619,1676077256908-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2023-02-11 01:00:57,761 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,46619,1676077256908-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2023-02-11 01:00:57,761 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,46619,1676077256908-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2023-02-11 01:00:57,761 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=CatalogJanitor-jenkins-hbase4:46619, period=300000, unit=MILLISECONDS is enabled. 2023-02-11 01:00:57,761 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2023-02-11 01:00:57,762 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.TableNamespaceManager(92): Namespace table not found. Creating... 2023-02-11 01:00:57,762 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2138): Client=null/null create 'hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-02-11 01:00:57,763 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2023-02-11 01:00:57,763 DEBUG [master/jenkins-hbase4:0.Chore.1] janitor.CatalogJanitor(175): 2023-02-11 01:00:57,764 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_PRE_OPERATION 2023-02-11 01:00:57,766 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-02-11 01:00:57,768 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:45625/user/jenkins/test-data/2a140438-d2b1-6994-16d5-13e6c7986734/.tmp/data/hbase/namespace/3b681ac3cacd3272c79e667e77ed237d 2023-02-11 01:00:57,768 DEBUG [HFileArchiver-2] backup.HFileArchiver(153): Directory hdfs://localhost:45625/user/jenkins/test-data/2a140438-d2b1-6994-16d5-13e6c7986734/.tmp/data/hbase/namespace/3b681ac3cacd3272c79e667e77ed237d empty. 2023-02-11 01:00:57,769 DEBUG [HFileArchiver-2] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:45625/user/jenkins/test-data/2a140438-d2b1-6994-16d5-13e6c7986734/.tmp/data/hbase/namespace/3b681ac3cacd3272c79e667e77ed237d 2023-02-11 01:00:57,769 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived hbase:namespace regions 2023-02-11 01:00:57,786 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:45625/user/jenkins/test-data/2a140438-d2b1-6994-16d5-13e6c7986734/.tmp/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 2023-02-11 01:00:57,788 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(7671): creating {ENCODED => 3b681ac3cacd3272c79e667e77ed237d, NAME => 'hbase:namespace,,1676077257762.3b681ac3cacd3272c79e667e77ed237d.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:45625/user/jenkins/test-data/2a140438-d2b1-6994-16d5-13e6c7986734/.tmp 2023-02-11 01:00:57,803 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(865): Instantiated hbase:namespace,,1676077257762.3b681ac3cacd3272c79e667e77ed237d.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 01:00:57,803 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1603): Closing 3b681ac3cacd3272c79e667e77ed237d, disabling compactions & flushes 2023-02-11 01:00:57,803 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1625): Closing region hbase:namespace,,1676077257762.3b681ac3cacd3272c79e667e77ed237d. 2023-02-11 01:00:57,803 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1646): Waiting without time limit for close lock on hbase:namespace,,1676077257762.3b681ac3cacd3272c79e667e77ed237d. 2023-02-11 01:00:57,803 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1713): Acquired close lock on hbase:namespace,,1676077257762.3b681ac3cacd3272c79e667e77ed237d. after waiting 0 ms 2023-02-11 01:00:57,803 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1723): Updates disabled for region hbase:namespace,,1676077257762.3b681ac3cacd3272c79e667e77ed237d. 2023-02-11 01:00:57,803 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1837): Closed hbase:namespace,,1676077257762.3b681ac3cacd3272c79e667e77ed237d. 2023-02-11 01:00:57,803 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1557): Region close journal for 3b681ac3cacd3272c79e667e77ed237d: Waiting for close lock at 1676077257803Disabling compacts and flushes for region at 1676077257803Disabling writes for close at 1676077257803Writing region close event to WAL at 1676077257803Closed at 1676077257803 2023-02-11 01:00:57,806 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ADD_TO_META 2023-02-11 01:00:57,808 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"hbase:namespace,,1676077257762.3b681ac3cacd3272c79e667e77ed237d.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1676077257808"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676077257808"}]},"ts":"1676077257808"} 2023-02-11 01:00:57,811 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-02-11 01:00:57,812 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-02-11 01:00:57,812 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676077257812"}]},"ts":"1676077257812"} 2023-02-11 01:00:57,814 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLING in hbase:meta 2023-02-11 01:00:57,818 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-02-11 01:00:57,819 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-11 01:00:57,819 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-11 01:00:57,819 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-11 01:00:57,819 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-11 01:00:57,819 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=3b681ac3cacd3272c79e667e77ed237d, ASSIGN}] 2023-02-11 01:00:57,821 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=3b681ac3cacd3272c79e667e77ed237d, ASSIGN 2023-02-11 01:00:57,823 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=3b681ac3cacd3272c79e667e77ed237d, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,43507,1676077257019; forceNewPlan=false, retain=false 2023-02-11 01:00:57,973 INFO [jenkins-hbase4:46619] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-02-11 01:00:57,974 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=3b681ac3cacd3272c79e667e77ed237d, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,43507,1676077257019 2023-02-11 01:00:57,974 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1676077257762.3b681ac3cacd3272c79e667e77ed237d.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1676077257974"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676077257974"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676077257974"}]},"ts":"1676077257974"} 2023-02-11 01:00:57,977 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=6, ppid=5, state=RUNNABLE; OpenRegionProcedure 3b681ac3cacd3272c79e667e77ed237d, server=jenkins-hbase4.apache.org,43507,1676077257019}] 2023-02-11 01:00:58,132 DEBUG [RSProcedureDispatcher-pool-1] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,43507,1676077257019 2023-02-11 01:00:58,132 DEBUG [RSProcedureDispatcher-pool-1] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-02-11 01:00:58,134 INFO [RS-EventLoopGroup-10-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:34882, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-02-11 01:00:58,139 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open hbase:namespace,,1676077257762.3b681ac3cacd3272c79e667e77ed237d. 2023-02-11 01:00:58,139 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 3b681ac3cacd3272c79e667e77ed237d, NAME => 'hbase:namespace,,1676077257762.3b681ac3cacd3272c79e667e77ed237d.', STARTKEY => '', ENDKEY => ''} 2023-02-11 01:00:58,139 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace 3b681ac3cacd3272c79e667e77ed237d 2023-02-11 01:00:58,139 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated hbase:namespace,,1676077257762.3b681ac3cacd3272c79e667e77ed237d.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-11 01:00:58,139 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 3b681ac3cacd3272c79e667e77ed237d 2023-02-11 01:00:58,139 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 3b681ac3cacd3272c79e667e77ed237d 2023-02-11 01:00:58,141 INFO [StoreOpener-3b681ac3cacd3272c79e667e77ed237d-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 3b681ac3cacd3272c79e667e77ed237d 2023-02-11 01:00:58,143 DEBUG [StoreOpener-3b681ac3cacd3272c79e667e77ed237d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45625/user/jenkins/test-data/2a140438-d2b1-6994-16d5-13e6c7986734/data/hbase/namespace/3b681ac3cacd3272c79e667e77ed237d/info 2023-02-11 01:00:58,143 DEBUG [StoreOpener-3b681ac3cacd3272c79e667e77ed237d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45625/user/jenkins/test-data/2a140438-d2b1-6994-16d5-13e6c7986734/data/hbase/namespace/3b681ac3cacd3272c79e667e77ed237d/info 2023-02-11 01:00:58,144 INFO [StoreOpener-3b681ac3cacd3272c79e667e77ed237d-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 3b681ac3cacd3272c79e667e77ed237d columnFamilyName info 2023-02-11 01:00:58,145 INFO [StoreOpener-3b681ac3cacd3272c79e667e77ed237d-1] regionserver.HStore(310): Store=3b681ac3cacd3272c79e667e77ed237d/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-11 01:00:58,146 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:45625/user/jenkins/test-data/2a140438-d2b1-6994-16d5-13e6c7986734/data/hbase/namespace/3b681ac3cacd3272c79e667e77ed237d 2023-02-11 01:00:58,147 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:45625/user/jenkins/test-data/2a140438-d2b1-6994-16d5-13e6c7986734/data/hbase/namespace/3b681ac3cacd3272c79e667e77ed237d 2023-02-11 01:00:58,150 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 3b681ac3cacd3272c79e667e77ed237d 2023-02-11 01:00:58,153 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45625/user/jenkins/test-data/2a140438-d2b1-6994-16d5-13e6c7986734/data/hbase/namespace/3b681ac3cacd3272c79e667e77ed237d/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-11 01:00:58,153 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 3b681ac3cacd3272c79e667e77ed237d; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=59754816, jitterRate=-0.10958385467529297}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-11 01:00:58,153 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 3b681ac3cacd3272c79e667e77ed237d: Running coprocessor pre-open hook at 1676077258140Writing region info on filesystem at 1676077258140Initializing all the Stores at 1676077258141 (+1 ms)Instantiating store for column family {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} at 1676077258141Cleaning up temporary data from old regions at 1676077258148 (+7 ms)Cleaning up detritus from prior splits at 1676077258148Running coprocessor post-open hooks at 1676077258153 (+5 ms)Region opened successfully at 1676077258153 2023-02-11 01:00:58,154 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:namespace,,1676077257762.3b681ac3cacd3272c79e667e77ed237d., pid=6, masterSystemTime=1676077258132 2023-02-11 01:00:58,160 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:namespace,,1676077257762.3b681ac3cacd3272c79e667e77ed237d. 2023-02-11 01:00:58,160 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened hbase:namespace,,1676077257762.3b681ac3cacd3272c79e667e77ed237d. 2023-02-11 01:00:58,161 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=3b681ac3cacd3272c79e667e77ed237d, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,43507,1676077257019 2023-02-11 01:00:58,162 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1676077257762.3b681ac3cacd3272c79e667e77ed237d.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1676077258161"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676077258161"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676077258161"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676077258161"}]},"ts":"1676077258161"} 2023-02-11 01:00:58,167 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=6, resume processing ppid=5 2023-02-11 01:00:58,167 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=6, ppid=5, state=SUCCESS; OpenRegionProcedure 3b681ac3cacd3272c79e667e77ed237d, server=jenkins-hbase4.apache.org,43507,1676077257019 in 187 msec 2023-02-11 01:00:58,170 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=5, resume processing ppid=4 2023-02-11 01:00:58,171 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=5, ppid=4, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=3b681ac3cacd3272c79e667e77ed237d, ASSIGN in 348 msec 2023-02-11 01:00:58,172 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-02-11 01:00:58,172 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676077258172"}]},"ts":"1676077258172"} 2023-02-11 01:00:58,174 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLED in hbase:meta 2023-02-11 01:00:58,176 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_POST_OPERATION 2023-02-11 01:00:58,178 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace in 414 msec 2023-02-11 01:00:58,264 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:46619-0x10040e53a700000, quorum=127.0.0.1:55753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/namespace 2023-02-11 01:00:58,265 DEBUG [Listener at localhost/43805-EventThread] zookeeper.ZKWatcher(600): master:46619-0x10040e53a700000, quorum=127.0.0.1:55753, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/namespace 2023-02-11 01:00:58,265 DEBUG [Listener at localhost/43805-EventThread] zookeeper.ZKWatcher(600): master:46619-0x10040e53a700000, quorum=127.0.0.1:55753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-11 01:00:58,268 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-11 01:00:58,270 INFO [RS-EventLoopGroup-10-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:34894, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-11 01:00:58,272 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=7, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2023-02-11 01:00:58,284 DEBUG [Listener at localhost/43805-EventThread] zookeeper.ZKWatcher(600): master:46619-0x10040e53a700000, quorum=127.0.0.1:55753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-02-11 01:00:58,288 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default in 15 msec 2023-02-11 01:00:58,294 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=8, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2023-02-11 01:00:58,303 DEBUG [Listener at localhost/43805-EventThread] zookeeper.ZKWatcher(600): master:46619-0x10040e53a700000, quorum=127.0.0.1:55753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-02-11 01:00:58,311 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase in 14 msec 2023-02-11 01:00:58,319 DEBUG [Listener at localhost/43805-EventThread] zookeeper.ZKWatcher(600): master:46619-0x10040e53a700000, quorum=127.0.0.1:55753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-02-11 01:00:58,321 DEBUG [Listener at localhost/43805-EventThread] zookeeper.ZKWatcher(600): master:46619-0x10040e53a700000, quorum=127.0.0.1:55753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2023-02-11 01:00:58,321 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1077): Master has completed initialization 1.265sec 2023-02-11 01:00:58,322 INFO [master/jenkins-hbase4:0:becomeActiveMaster] quotas.MasterQuotaManager(97): Quota support disabled 2023-02-11 01:00:58,322 INFO [master/jenkins-hbase4:0:becomeActiveMaster] slowlog.SlowLogMasterService(57): Slow/Large requests logging to system table hbase:slowlog is disabled. Quitting. 2023-02-11 01:00:58,322 INFO [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKWatcher(269): not a secure deployment, proceeding 2023-02-11 01:00:58,322 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,46619,1676077256908-ExpiredMobFileCleanerChore, period=86400, unit=SECONDS is enabled. 2023-02-11 01:00:58,322 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,46619,1676077256908-MobCompactionChore, period=604800, unit=SECONDS is enabled. 2023-02-11 01:00:58,324 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1166): Balancer post startup initialization complete, took 0 seconds 2023-02-11 01:00:58,349 DEBUG [Listener at localhost/43805] zookeeper.ReadOnlyZKClient(139): Connect 0x7284ed35 to 127.0.0.1:55753 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-11 01:00:58,355 DEBUG [Listener at localhost/43805] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@17d17331, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-11 01:00:58,357 DEBUG [hconnection-0x13815be5-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-11 01:00:58,359 INFO [RS-EventLoopGroup-9-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39750, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-11 01:00:58,361 INFO [Listener at localhost/43805] hbase.HBaseTestingUtility(1145): Minicluster is up; activeMaster=jenkins-hbase4.apache.org,46619,1676077256908 2023-02-11 01:00:58,361 DEBUG [Listener at localhost/43805] zookeeper.ReadOnlyZKClient(139): Connect 0x33280b48 to 127.0.0.1:55753 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-11 01:00:58,367 DEBUG [ReadOnlyZKClient-127.0.0.1:55753@0x33280b48] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5310f02e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-11 01:00:58,368 DEBUG [Listener at localhost/43805] client.ConnectionUtils(586): Start fetching master stub from registry 2023-02-11 01:00:58,369 DEBUG [ReadOnlyZKClient-127.0.0.1:55753@0x33280b48] client.AsyncConnectionImpl(289): The fetched master address is jenkins-hbase4.apache.org,46619,1676077256908 2023-02-11 01:00:58,369 DEBUG [ReadOnlyZKClient-127.0.0.1:55753@0x33280b48] client.ConnectionUtils(594): The fetched master stub is org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$Stub@572f6e94 2023-02-11 01:00:58,372 DEBUG [ReadOnlyZKClient-127.0.0.1:55753@0x33280b48] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-02-11 01:00:58,374 INFO [RS-EventLoopGroup-7-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59838, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-02-11 01:00:58,374 INFO [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=46619] master.MasterRpcServices(1560): Client=jenkins//172.31.14.131 shutdown 2023-02-11 01:00:58,374 INFO [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=46619] master.ServerManager(901): Cluster shutdown requested of master=jenkins-hbase4.apache.org,46619,1676077256908 2023-02-11 01:00:58,376 DEBUG [Listener at localhost/43805-EventThread] zookeeper.ZKWatcher(600): regionserver:43507-0x10040e53a700003, quorum=127.0.0.1:55753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-02-11 01:00:58,376 DEBUG [Listener at localhost/43805-EventThread] zookeeper.ZKWatcher(600): master:46619-0x10040e53a700000, quorum=127.0.0.1:55753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-02-11 01:00:58,376 INFO [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=46619] procedure2.ProcedureExecutor(629): Stopping 2023-02-11 01:00:58,376 DEBUG [Listener at localhost/43805-EventThread] zookeeper.ZKWatcher(600): master:46619-0x10040e53a700000, quorum=127.0.0.1:55753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-11 01:00:58,376 DEBUG [Listener at localhost/43805-EventThread] zookeeper.ZKWatcher(600): regionserver:36543-0x10040e53a700001, quorum=127.0.0.1:55753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-02-11 01:00:58,376 DEBUG [Listener at localhost/43805-EventThread] zookeeper.ZKWatcher(600): regionserver:38923-0x10040e53a700002, quorum=127.0.0.1:55753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-02-11 01:00:58,377 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:43507-0x10040e53a700003, quorum=127.0.0.1:55753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-11 01:00:58,377 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:46619-0x10040e53a700000, quorum=127.0.0.1:55753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-11 01:00:58,377 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=46619] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x68d75474 to 127.0.0.1:55753 2023-02-11 01:00:58,377 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36543-0x10040e53a700001, quorum=127.0.0.1:55753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-11 01:00:58,377 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=46619] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-11 01:00:58,377 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38923-0x10040e53a700002, quorum=127.0.0.1:55753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-11 01:00:58,441 INFO [RS:2;jenkins-hbase4:43507] regionserver.HRegionServer(1065): Closing user regions 2023-02-11 01:00:58,441 INFO [RS:2;jenkins-hbase4:43507] regionserver.HRegionServer(3304): Received CLOSE for 3b681ac3cacd3272c79e667e77ed237d 2023-02-11 01:00:58,442 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 3b681ac3cacd3272c79e667e77ed237d, disabling compactions & flushes 2023-02-11 01:00:58,442 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region hbase:namespace,,1676077257762.3b681ac3cacd3272c79e667e77ed237d. 2023-02-11 01:00:58,442 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on hbase:namespace,,1676077257762.3b681ac3cacd3272c79e667e77ed237d. 2023-02-11 01:00:58,442 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on hbase:namespace,,1676077257762.3b681ac3cacd3272c79e667e77ed237d. after waiting 0 ms 2023-02-11 01:00:58,442 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region hbase:namespace,,1676077257762.3b681ac3cacd3272c79e667e77ed237d. 2023-02-11 01:00:58,442 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 3b681ac3cacd3272c79e667e77ed237d 1/1 column families, dataSize=78 B heapSize=488 B 2023-02-11 01:00:58,447 INFO [RS:1;jenkins-hbase4:38923] regionserver.HRegionServer(1065): Closing user regions 2023-02-11 01:00:58,453 INFO [RS:0;jenkins-hbase4:36543] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,36543,1676077256955' ***** 2023-02-11 01:00:58,453 INFO [RS:0;jenkins-hbase4:36543] regionserver.HRegionServer(2310): STOPPED: Exiting; cluster shutdown set and not carrying any regions 2023-02-11 01:00:58,455 INFO [RS:0;jenkins-hbase4:36543] regionserver.HeapMemoryManager(220): Stopping 2023-02-11 01:00:58,456 INFO [RS:0;jenkins-hbase4:36543] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-02-11 01:00:58,456 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-02-11 01:00:58,456 INFO [RS:0;jenkins-hbase4:36543] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-02-11 01:00:58,456 INFO [RS:0;jenkins-hbase4:36543] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,36543,1676077256955 2023-02-11 01:00:58,456 INFO [regionserver/jenkins-hbase4:0.Chore.1] hbase.ScheduledChore(146): Chore: CompactionChecker was stopped 2023-02-11 01:00:58,456 DEBUG [RS:0;jenkins-hbase4:36543] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6a511774 to 127.0.0.1:55753 2023-02-11 01:00:58,457 INFO [regionserver/jenkins-hbase4:0.Chore.1] hbase.ScheduledChore(146): Chore: MemstoreFlusherChore was stopped 2023-02-11 01:00:58,457 DEBUG [RS:0;jenkins-hbase4:36543] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-11 01:00:58,457 INFO [RS:0;jenkins-hbase4:36543] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,36543,1676077256955; all regions closed. 2023-02-11 01:00:58,464 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=78 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost:45625/user/jenkins/test-data/2a140438-d2b1-6994-16d5-13e6c7986734/data/hbase/namespace/3b681ac3cacd3272c79e667e77ed237d/.tmp/info/34306d16d9b54adaa7664bc67faf3cd6 2023-02-11 01:00:58,466 DEBUG [RS:0;jenkins-hbase4:36543] wal.AbstractFSWAL(932): Moved 1 WAL file(s) to /user/jenkins/test-data/2a140438-d2b1-6994-16d5-13e6c7986734/oldWALs 2023-02-11 01:00:58,466 INFO [RS:0;jenkins-hbase4:36543] wal.AbstractFSWAL(935): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C36543%2C1676077256955:(num 1676077257514) 2023-02-11 01:00:58,466 DEBUG [RS:0;jenkins-hbase4:36543] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-11 01:00:58,466 INFO [RS:0;jenkins-hbase4:36543] regionserver.LeaseManager(133): Closed leases 2023-02-11 01:00:58,466 INFO [RS:0;jenkins-hbase4:36543] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS, ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS] on shutdown 2023-02-11 01:00:58,467 INFO [RS:0;jenkins-hbase4:36543] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-02-11 01:00:58,467 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-02-11 01:00:58,467 INFO [RS:0;jenkins-hbase4:36543] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-02-11 01:00:58,467 INFO [RS:0;jenkins-hbase4:36543] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-02-11 01:00:58,467 INFO [RS:0;jenkins-hbase4:36543] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:36543 2023-02-11 01:00:58,476 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-02-11 01:00:58,479 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45625/user/jenkins/test-data/2a140438-d2b1-6994-16d5-13e6c7986734/data/hbase/namespace/3b681ac3cacd3272c79e667e77ed237d/.tmp/info/34306d16d9b54adaa7664bc67faf3cd6 as hdfs://localhost:45625/user/jenkins/test-data/2a140438-d2b1-6994-16d5-13e6c7986734/data/hbase/namespace/3b681ac3cacd3272c79e667e77ed237d/info/34306d16d9b54adaa7664bc67faf3cd6 2023-02-11 01:00:58,487 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:45625/user/jenkins/test-data/2a140438-d2b1-6994-16d5-13e6c7986734/data/hbase/namespace/3b681ac3cacd3272c79e667e77ed237d/info/34306d16d9b54adaa7664bc67faf3cd6, entries=2, sequenceid=6, filesize=4.8 K 2023-02-11 01:00:58,489 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~78 B/78, heapSize ~472 B/472, currentSize=0 B/0 for 3b681ac3cacd3272c79e667e77ed237d in 47ms, sequenceid=6, compaction requested=false 2023-02-11 01:00:58,497 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45625/user/jenkins/test-data/2a140438-d2b1-6994-16d5-13e6c7986734/data/hbase/namespace/3b681ac3cacd3272c79e667e77ed237d/recovered.edits/9.seqid, newMaxSeqId=9, maxSeqId=1 2023-02-11 01:00:58,498 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed hbase:namespace,,1676077257762.3b681ac3cacd3272c79e667e77ed237d. 2023-02-11 01:00:58,498 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 3b681ac3cacd3272c79e667e77ed237d: Waiting for close lock at 1676077258442Running coprocessor pre-close hooks at 1676077258442Disabling compacts and flushes for region at 1676077258442Disabling writes for close at 1676077258442Obtaining lock to block concurrent updates at 1676077258442Preparing flush snapshotting stores in 3b681ac3cacd3272c79e667e77ed237d at 1676077258442Finished memstore snapshotting hbase:namespace,,1676077257762.3b681ac3cacd3272c79e667e77ed237d., syncing WAL and waiting on mvcc, flushsize=dataSize=78, getHeapSize=472, getOffHeapSize=0, getCellsCount=2 at 1676077258443 (+1 ms)Flushing stores of hbase:namespace,,1676077257762.3b681ac3cacd3272c79e667e77ed237d. at 1676077258444 (+1 ms)Flushing 3b681ac3cacd3272c79e667e77ed237d/info: creating writer at 1676077258444Flushing 3b681ac3cacd3272c79e667e77ed237d/info: appending metadata at 1676077258447 (+3 ms)Flushing 3b681ac3cacd3272c79e667e77ed237d/info: closing flushed file at 1676077258447Flushing 3b681ac3cacd3272c79e667e77ed237d/info: reopening flushed file at 1676077258480 (+33 ms)Finished flush of dataSize ~78 B/78, heapSize ~472 B/472, currentSize=0 B/0 for 3b681ac3cacd3272c79e667e77ed237d in 47ms, sequenceid=6, compaction requested=false at 1676077258489 (+9 ms)Writing region close event to WAL at 1676077258491 (+2 ms)Running coprocessor post-close hooks at 1676077258498 (+7 ms)Closed at 1676077258498 2023-02-11 01:00:58,499 INFO [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=46619] assignment.AssignmentManager(1094): RegionServer CLOSED 3b681ac3cacd3272c79e667e77ed237d 2023-02-11 01:00:58,500 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1676077257762.3b681ac3cacd3272c79e667e77ed237d. 2023-02-11 01:00:58,544 INFO [RS:2;jenkins-hbase4:43507] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,43507,1676077257019' ***** 2023-02-11 01:00:58,544 INFO [RS:2;jenkins-hbase4:43507] regionserver.HRegionServer(2310): STOPPED: Exiting; cluster shutdown set and not carrying any regions 2023-02-11 01:00:58,545 INFO [RS:2;jenkins-hbase4:43507] regionserver.HeapMemoryManager(220): Stopping 2023-02-11 01:00:58,545 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-02-11 01:00:58,545 INFO [RS:2;jenkins-hbase4:43507] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-02-11 01:00:58,546 INFO [RS:2;jenkins-hbase4:43507] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-02-11 01:00:58,546 INFO [RS:2;jenkins-hbase4:43507] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,43507,1676077257019 2023-02-11 01:00:58,546 DEBUG [RS:2;jenkins-hbase4:43507] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0fffc065 to 127.0.0.1:55753 2023-02-11 01:00:58,546 DEBUG [RS:2;jenkins-hbase4:43507] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-11 01:00:58,546 INFO [RS:2;jenkins-hbase4:43507] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,43507,1676077257019; all regions closed. 2023-02-11 01:00:58,548 DEBUG [RS:1;jenkins-hbase4:38923] regionserver.HRegionServer(1084): Waiting on 1588230740 2023-02-11 01:00:58,549 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-02-11 01:00:58,553 DEBUG [RS:2;jenkins-hbase4:43507] wal.AbstractFSWAL(932): Moved 1 WAL file(s) to /user/jenkins/test-data/2a140438-d2b1-6994-16d5-13e6c7986734/oldWALs 2023-02-11 01:00:58,553 INFO [RS:2;jenkins-hbase4:43507] wal.AbstractFSWAL(935): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C43507%2C1676077257019:(num 1676077257489) 2023-02-11 01:00:58,554 DEBUG [RS:2;jenkins-hbase4:43507] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-11 01:00:58,554 INFO [RS:2;jenkins-hbase4:43507] regionserver.LeaseManager(133): Closed leases 2023-02-11 01:00:58,554 INFO [RS:2;jenkins-hbase4:43507] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS, ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS] on shutdown 2023-02-11 01:00:58,554 INFO [RS:2;jenkins-hbase4:43507] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-02-11 01:00:58,554 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-02-11 01:00:58,554 INFO [RS:2;jenkins-hbase4:43507] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-02-11 01:00:58,554 INFO [RS:2;jenkins-hbase4:43507] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-02-11 01:00:58,555 INFO [RS:2;jenkins-hbase4:43507] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:43507 2023-02-11 01:00:58,564 DEBUG [Listener at localhost/43805-EventThread] zookeeper.ZKWatcher(600): regionserver:36543-0x10040e53a700001, quorum=127.0.0.1:55753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,36543,1676077256955 2023-02-11 01:00:58,564 DEBUG [Listener at localhost/43805-EventThread] zookeeper.ZKWatcher(600): master:46619-0x10040e53a700000, quorum=127.0.0.1:55753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-11 01:00:58,564 DEBUG [Listener at localhost/43805-EventThread] zookeeper.ZKWatcher(600): regionserver:36543-0x10040e53a700001, quorum=127.0.0.1:55753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-11 01:00:58,564 DEBUG [Listener at localhost/43805-EventThread] zookeeper.ZKWatcher(600): regionserver:38923-0x10040e53a700002, quorum=127.0.0.1:55753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,36543,1676077256955 2023-02-11 01:00:58,564 DEBUG [Listener at localhost/43805-EventThread] zookeeper.ZKWatcher(600): regionserver:43507-0x10040e53a700003, quorum=127.0.0.1:55753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,36543,1676077256955 2023-02-11 01:00:58,564 DEBUG [Listener at localhost/43805-EventThread] zookeeper.ZKWatcher(600): regionserver:43507-0x10040e53a700003, quorum=127.0.0.1:55753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-11 01:00:58,564 DEBUG [Listener at localhost/43805-EventThread] zookeeper.ZKWatcher(600): regionserver:38923-0x10040e53a700002, quorum=127.0.0.1:55753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-11 01:00:58,565 DEBUG [Listener at localhost/43805-EventThread] zookeeper.ZKWatcher(600): regionserver:36543-0x10040e53a700001, quorum=127.0.0.1:55753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,43507,1676077257019 2023-02-11 01:00:58,565 ERROR [Listener at localhost/43805-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@285db293 rejected from java.util.concurrent.ThreadPoolExecutor@2c622340[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 6] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-11 01:00:58,566 DEBUG [Listener at localhost/43805-EventThread] zookeeper.ZKWatcher(600): regionserver:38923-0x10040e53a700002, quorum=127.0.0.1:55753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,43507,1676077257019 2023-02-11 01:00:58,565 DEBUG [Listener at localhost/43805-EventThread] zookeeper.ZKWatcher(600): regionserver:43507-0x10040e53a700003, quorum=127.0.0.1:55753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,43507,1676077257019 2023-02-11 01:00:58,565 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase4.apache.org,43507,1676077257019] 2023-02-11 01:00:58,566 ERROR [Listener at localhost/43805-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@d4e1ca9 rejected from java.util.concurrent.ThreadPoolExecutor@97199d8[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 6] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-11 01:00:58,566 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase4.apache.org,43507,1676077257019; numProcessing=1 2023-02-11 01:00:58,568 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:38923-0x10040e53a700002, quorum=127.0.0.1:55753, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,38923,1676077256985 2023-02-11 01:00:58,568 INFO [zk-event-processor-pool-0] replication.ReplicationTrackerZKImpl$OtherRegionServerWatcher(124): /hbase/rs/jenkins-hbase4.apache.org,36543,1676077256955 znode expired, triggering replicatorRemoved event 2023-02-11 01:00:58,569 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,43507,1676077257019 already deleted, retry=false 2023-02-11 01:00:58,569 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase4.apache.org,43507,1676077257019 expired; onlineServers=2 2023-02-11 01:00:58,569 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase4.apache.org,36543,1676077256955] 2023-02-11 01:00:58,569 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase4.apache.org,36543,1676077256955; numProcessing=2 2023-02-11 01:00:58,569 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:38923-0x10040e53a700002, quorum=127.0.0.1:55753, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,38923,1676077256985 2023-02-11 01:00:58,570 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,36543,1676077256955 already deleted, retry=false 2023-02-11 01:00:58,570 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase4.apache.org,36543,1676077256955 expired; onlineServers=1 2023-02-11 01:00:58,570 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): regionserver:38923-0x10040e53a700002, quorum=127.0.0.1:55753, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,38923,1676077256985 2023-02-11 01:00:58,570 INFO [zk-event-processor-pool-0] replication.ReplicationTrackerZKImpl$OtherRegionServerWatcher(124): /hbase/rs/jenkins-hbase4.apache.org,43507,1676077257019 znode expired, triggering replicatorRemoved event 2023-02-11 01:00:58,651 INFO [RS:1;jenkins-hbase4:38923] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,38923,1676077256985' ***** 2023-02-11 01:00:58,651 INFO [RS:1;jenkins-hbase4:38923] regionserver.HRegionServer(2310): STOPPED: Stopped; only catalog regions remaining online 2023-02-11 01:00:58,652 INFO [RS:1;jenkins-hbase4:38923] regionserver.HeapMemoryManager(220): Stopping 2023-02-11 01:00:58,652 INFO [RS:1;jenkins-hbase4:38923] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-02-11 01:00:58,652 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-02-11 01:00:58,652 INFO [RS:1;jenkins-hbase4:38923] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-02-11 01:00:58,652 INFO [RS:1;jenkins-hbase4:38923] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,38923,1676077256985 2023-02-11 01:00:58,652 DEBUG [RS:1;jenkins-hbase4:38923] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2cd7531e to 127.0.0.1:55753 2023-02-11 01:00:58,653 DEBUG [RS:1;jenkins-hbase4:38923] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-11 01:00:58,653 INFO [RS:1;jenkins-hbase4:38923] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-02-11 01:00:58,653 INFO [RS:1;jenkins-hbase4:38923] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-02-11 01:00:58,653 INFO [RS:1;jenkins-hbase4:38923] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-02-11 01:00:58,653 INFO [RS:1;jenkins-hbase4:38923] regionserver.HRegionServer(3304): Received CLOSE for 1588230740 2023-02-11 01:00:58,653 INFO [RS:1;jenkins-hbase4:38923] regionserver.HRegionServer(1475): Waiting on 1 regions to close 2023-02-11 01:00:58,653 DEBUG [RS:1;jenkins-hbase4:38923] regionserver.HRegionServer(1479): Online Regions={1588230740=hbase:meta,,1.1588230740} 2023-02-11 01:00:58,654 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 1588230740, disabling compactions & flushes 2023-02-11 01:00:58,654 DEBUG [RS:1;jenkins-hbase4:38923] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-02-11 01:00:58,654 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region hbase:meta,,1.1588230740 2023-02-11 01:00:58,654 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-02-11 01:00:58,654 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-02-11 01:00:58,654 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region hbase:meta,,1.1588230740 2023-02-11 01:00:58,654 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 1588230740 3/3 column families, dataSize=1.26 KB heapSize=2.89 KB 2023-02-11 01:00:58,666 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-02-11 01:00:58,671 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=1.17 KB at sequenceid=9 (bloomFilter=false), to=hdfs://localhost:45625/user/jenkins/test-data/2a140438-d2b1-6994-16d5-13e6c7986734/data/hbase/meta/1588230740/.tmp/info/a6a7f35a96da4044891d275f4c4f5eb0 2023-02-11 01:00:58,689 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=94 B at sequenceid=9 (bloomFilter=false), to=hdfs://localhost:45625/user/jenkins/test-data/2a140438-d2b1-6994-16d5-13e6c7986734/data/hbase/meta/1588230740/.tmp/table/003d295d0ccf40b09014c596013a4088 2023-02-11 01:00:58,697 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45625/user/jenkins/test-data/2a140438-d2b1-6994-16d5-13e6c7986734/data/hbase/meta/1588230740/.tmp/info/a6a7f35a96da4044891d275f4c4f5eb0 as hdfs://localhost:45625/user/jenkins/test-data/2a140438-d2b1-6994-16d5-13e6c7986734/data/hbase/meta/1588230740/info/a6a7f35a96da4044891d275f4c4f5eb0 2023-02-11 01:00:58,704 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:45625/user/jenkins/test-data/2a140438-d2b1-6994-16d5-13e6c7986734/data/hbase/meta/1588230740/info/a6a7f35a96da4044891d275f4c4f5eb0, entries=10, sequenceid=9, filesize=5.9 K 2023-02-11 01:00:58,705 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45625/user/jenkins/test-data/2a140438-d2b1-6994-16d5-13e6c7986734/data/hbase/meta/1588230740/.tmp/table/003d295d0ccf40b09014c596013a4088 as hdfs://localhost:45625/user/jenkins/test-data/2a140438-d2b1-6994-16d5-13e6c7986734/data/hbase/meta/1588230740/table/003d295d0ccf40b09014c596013a4088 2023-02-11 01:00:58,719 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:45625/user/jenkins/test-data/2a140438-d2b1-6994-16d5-13e6c7986734/data/hbase/meta/1588230740/table/003d295d0ccf40b09014c596013a4088, entries=2, sequenceid=9, filesize=4.7 K 2023-02-11 01:00:58,720 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~1.26 KB/1290, heapSize ~2.61 KB/2672, currentSize=0 B/0 for 1588230740 in 66ms, sequenceid=9, compaction requested=false 2023-02-11 01:00:58,727 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45625/user/jenkins/test-data/2a140438-d2b1-6994-16d5-13e6c7986734/data/hbase/meta/1588230740/recovered.edits/12.seqid, newMaxSeqId=12, maxSeqId=1 2023-02-11 01:00:58,727 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-02-11 01:00:58,728 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed hbase:meta,,1.1588230740 2023-02-11 01:00:58,728 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 1588230740: Waiting for close lock at 1676077258653Running coprocessor pre-close hooks at 1676077258653Disabling compacts and flushes for region at 1676077258653Disabling writes for close at 1676077258654 (+1 ms)Obtaining lock to block concurrent updates at 1676077258654Preparing flush snapshotting stores in 1588230740 at 1676077258654Finished memstore snapshotting hbase:meta,,1.1588230740, syncing WAL and waiting on mvcc, flushsize=dataSize=1290, getHeapSize=2912, getOffHeapSize=0, getCellsCount=12 at 1676077258654Flushing stores of hbase:meta,,1.1588230740 at 1676077258655 (+1 ms)Flushing 1588230740/info: creating writer at 1676077258655Flushing 1588230740/info: appending metadata at 1676077258659 (+4 ms)Flushing 1588230740/info: closing flushed file at 1676077258659Flushing 1588230740/table: creating writer at 1676077258677 (+18 ms)Flushing 1588230740/table: appending metadata at 1676077258680 (+3 ms)Flushing 1588230740/table: closing flushed file at 1676077258680Flushing 1588230740/info: reopening flushed file at 1676077258698 (+18 ms)Flushing 1588230740/table: reopening flushed file at 1676077258706 (+8 ms)Finished flush of dataSize ~1.26 KB/1290, heapSize ~2.61 KB/2672, currentSize=0 B/0 for 1588230740 in 66ms, sequenceid=9, compaction requested=false at 1676077258720 (+14 ms)Writing region close event to WAL at 1676077258723 (+3 ms)Running coprocessor post-close hooks at 1676077258727 (+4 ms)Closed at 1676077258728 (+1 ms) 2023-02-11 01:00:58,728 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-02-11 01:00:58,854 INFO [RS:1;jenkins-hbase4:38923] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,38923,1676077256985; all regions closed. 2023-02-11 01:00:58,860 DEBUG [RS:1;jenkins-hbase4:38923] wal.AbstractFSWAL(932): Moved 1 WAL file(s) to /user/jenkins/test-data/2a140438-d2b1-6994-16d5-13e6c7986734/oldWALs 2023-02-11 01:00:58,861 INFO [RS:1;jenkins-hbase4:38923] wal.AbstractFSWAL(935): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C38923%2C1676077256985.meta:.meta(num 1676077257692) 2023-02-11 01:00:58,866 DEBUG [RS:1;jenkins-hbase4:38923] wal.AbstractFSWAL(932): Moved 1 WAL file(s) to /user/jenkins/test-data/2a140438-d2b1-6994-16d5-13e6c7986734/oldWALs 2023-02-11 01:00:58,866 INFO [RS:1;jenkins-hbase4:38923] wal.AbstractFSWAL(935): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C38923%2C1676077256985:(num 1676077257497) 2023-02-11 01:00:58,866 DEBUG [RS:1;jenkins-hbase4:38923] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-11 01:00:58,866 INFO [RS:1;jenkins-hbase4:38923] regionserver.LeaseManager(133): Closed leases 2023-02-11 01:00:58,866 DEBUG [Listener at localhost/43805-EventThread] zookeeper.ZKWatcher(600): regionserver:43507-0x10040e53a700003, quorum=127.0.0.1:55753, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-11 01:00:58,866 INFO [RS:2;jenkins-hbase4:43507] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,43507,1676077257019; zookeeper connection closed. 2023-02-11 01:00:58,866 INFO [RS:1;jenkins-hbase4:38923] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS, ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS] on shutdown 2023-02-11 01:00:58,866 ERROR [Listener at localhost/43805-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@a3b106 rejected from java.util.concurrent.ThreadPoolExecutor@97199d8[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 6] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-11 01:00:58,867 INFO [RS:1;jenkins-hbase4:38923] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:38923 2023-02-11 01:00:58,867 DEBUG [Listener at localhost/43805-EventThread] zookeeper.ZKWatcher(600): regionserver:43507-0x10040e53a700003, quorum=127.0.0.1:55753, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-11 01:00:58,867 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-02-11 01:00:58,867 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@9c3112d] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@9c3112d 2023-02-11 01:00:58,867 ERROR [Listener at localhost/43805-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@784fa7c0 rejected from java.util.concurrent.ThreadPoolExecutor@97199d8[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 6] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-11 01:00:58,872 DEBUG [Listener at localhost/43805-EventThread] zookeeper.ZKWatcher(600): regionserver:38923-0x10040e53a700002, quorum=127.0.0.1:55753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,38923,1676077256985 2023-02-11 01:00:58,872 DEBUG [Listener at localhost/43805-EventThread] zookeeper.ZKWatcher(600): master:46619-0x10040e53a700000, quorum=127.0.0.1:55753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-11 01:00:58,872 ERROR [Listener at localhost/43805-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@640ef8be rejected from java.util.concurrent.ThreadPoolExecutor@603170e7[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 7] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-11 01:00:58,873 DEBUG [Listener at localhost/43805-EventThread] zookeeper.ZKWatcher(600): regionserver:38923-0x10040e53a700002, quorum=127.0.0.1:55753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-11 01:00:58,873 ERROR [Listener at localhost/43805-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@7844017 rejected from java.util.concurrent.ThreadPoolExecutor@603170e7[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 7] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-11 01:00:58,873 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase4.apache.org,38923,1676077256985] 2023-02-11 01:00:58,873 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase4.apache.org,38923,1676077256985; numProcessing=3 2023-02-11 01:00:58,875 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,38923,1676077256985 already deleted, retry=false 2023-02-11 01:00:58,875 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase4.apache.org,38923,1676077256985 expired; onlineServers=0 2023-02-11 01:00:58,875 INFO [RegionServerTracker-0] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,46619,1676077256908' ***** 2023-02-11 01:00:58,875 INFO [RegionServerTracker-0] regionserver.HRegionServer(2310): STOPPED: Cluster shutdown set; onlineServer=0 2023-02-11 01:00:58,875 DEBUG [M:0;jenkins-hbase4:46619] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2bb6520e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-02-11 01:00:58,875 INFO [M:0;jenkins-hbase4:46619] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,46619,1676077256908 2023-02-11 01:00:58,875 INFO [M:0;jenkins-hbase4:46619] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,46619,1676077256908; all regions closed. 2023-02-11 01:00:58,875 DEBUG [M:0;jenkins-hbase4:46619] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-11 01:00:58,875 DEBUG [M:0;jenkins-hbase4:46619] cleaner.LogCleaner(198): Cancelling LogCleaner 2023-02-11 01:00:58,875 WARN [OldWALsCleaner-0] cleaner.LogCleaner(186): Interrupted while cleaning old WALs, will try to clean it next round. Exiting. 2023-02-11 01:00:58,875 DEBUG [M:0;jenkins-hbase4:46619] cleaner.HFileCleaner(317): Stopping file delete threads 2023-02-11 01:00:58,875 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1676077257231] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1676077257231,5,FailOnTimeoutGroup] 2023-02-11 01:00:58,875 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1676077257232] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1676077257232,5,FailOnTimeoutGroup] 2023-02-11 01:00:58,876 INFO [M:0;jenkins-hbase4:46619] master.MasterMobCompactionThread(168): Waiting for Mob Compaction Thread to finish... 2023-02-11 01:00:58,877 INFO [M:0;jenkins-hbase4:46619] master.MasterMobCompactionThread(168): Waiting for Region Server Mob Compaction Thread to finish... 2023-02-11 01:00:58,877 INFO [M:0;jenkins-hbase4:46619] hbase.ChoreService(369): Chore service for: master/jenkins-hbase4:0 had [] on shutdown 2023-02-11 01:00:58,877 DEBUG [Listener at localhost/43805-EventThread] zookeeper.ZKWatcher(600): master:46619-0x10040e53a700000, quorum=127.0.0.1:55753, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-02-11 01:00:58,877 DEBUG [M:0;jenkins-hbase4:46619] master.HMaster(1502): Stopping service threads 2023-02-11 01:00:58,877 DEBUG [Listener at localhost/43805-EventThread] zookeeper.ZKWatcher(600): master:46619-0x10040e53a700000, quorum=127.0.0.1:55753, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-11 01:00:58,877 INFO [M:0;jenkins-hbase4:46619] procedure2.RemoteProcedureDispatcher(118): Stopping procedure remote dispatcher 2023-02-11 01:00:58,877 ERROR [M:0;jenkins-hbase4:46619] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] Thread[HFileArchiver-2,5,PEWorkerGroup] 2023-02-11 01:00:58,877 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:46619-0x10040e53a700000, quorum=127.0.0.1:55753, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-11 01:00:58,878 INFO [M:0;jenkins-hbase4:46619] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-02-11 01:00:58,878 DEBUG [normalizer-worker-0] normalizer.RegionNormalizerWorker(174): interrupt detected. terminating. 2023-02-11 01:00:58,878 DEBUG [M:0;jenkins-hbase4:46619] zookeeper.ZKUtil(398): master:46619-0x10040e53a700000, quorum=127.0.0.1:55753, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2023-02-11 01:00:58,878 WARN [M:0;jenkins-hbase4:46619] master.ActiveMasterManager(323): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2023-02-11 01:00:58,878 INFO [M:0;jenkins-hbase4:46619] assignment.AssignmentManager(315): Stopping assignment manager 2023-02-11 01:00:58,878 INFO [M:0;jenkins-hbase4:46619] region.MasterRegion(167): Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=false 2023-02-11 01:00:58,879 DEBUG [M:0;jenkins-hbase4:46619] regionserver.HRegion(1603): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-02-11 01:00:58,879 INFO [M:0;jenkins-hbase4:46619] regionserver.HRegion(1625): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-11 01:00:58,879 DEBUG [M:0;jenkins-hbase4:46619] regionserver.HRegion(1646): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-11 01:00:58,879 DEBUG [M:0;jenkins-hbase4:46619] regionserver.HRegion(1713): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-02-11 01:00:58,879 DEBUG [M:0;jenkins-hbase4:46619] regionserver.HRegion(1723): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-11 01:00:58,879 INFO [M:0;jenkins-hbase4:46619] regionserver.HRegion(2744): Flushing 1595e783b53d99cd5eef43b6debb2682 1/1 column families, dataSize=24.07 KB heapSize=29.55 KB 2023-02-11 01:00:58,880 INFO [Listener at localhost/43805] client.AsyncConnectionImpl(207): Connection has been closed by Listener at localhost/43805. 2023-02-11 01:00:58,880 DEBUG [Listener at localhost/43805] client.AsyncConnectionImpl(232): Call stack: at java.lang.Thread.getStackTrace(Thread.java:1564) at org.apache.hadoop.hbase.client.AsyncConnectionImpl.close(AsyncConnectionImpl.java:209) at org.apache.hbase.thirdparty.com.google.common.io.Closeables.close(Closeables.java:79) at org.apache.hadoop.hbase.client.TestAsyncClusterAdminApi2.tearDown(TestAsyncClusterAdminApi2.java:75) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56) at org.junit.internal.runners.statements.RunAfters.invokeMethod(RunAfters.java:46) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:33) at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61) at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306) at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63) at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329) at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.apache.hadoop.hbase.SystemExitRule$1.evaluate(SystemExitRule.java:39) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:750) 2023-02-11 01:00:58,880 DEBUG [Listener at localhost/43805] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-11 01:00:58,881 DEBUG [Listener at localhost/43805] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x33280b48 to 127.0.0.1:55753 2023-02-11 01:00:58,881 INFO [Listener at localhost/43805] hbase.HBaseTestingUtility(1286): Shutting down minicluster 2023-02-11 01:00:58,882 DEBUG [Listener at localhost/43805] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7284ed35 to 127.0.0.1:55753 2023-02-11 01:00:58,882 DEBUG [Listener at localhost/43805] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-11 01:00:58,882 DEBUG [Listener at localhost/43805] util.JVMClusterUtil(237): Shutting down HBase Cluster 2023-02-11 01:00:58,896 INFO [M:0;jenkins-hbase4:46619] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.07 KB at sequenceid=66 (bloomFilter=true), to=hdfs://localhost:45625/user/jenkins/test-data/2a140438-d2b1-6994-16d5-13e6c7986734/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/5208485b8bc04aabad9406fd26f01e78 2023-02-11 01:00:58,903 DEBUG [M:0;jenkins-hbase4:46619] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45625/user/jenkins/test-data/2a140438-d2b1-6994-16d5-13e6c7986734/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/5208485b8bc04aabad9406fd26f01e78 as hdfs://localhost:45625/user/jenkins/test-data/2a140438-d2b1-6994-16d5-13e6c7986734/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/5208485b8bc04aabad9406fd26f01e78 2023-02-11 01:00:58,909 INFO [M:0;jenkins-hbase4:46619] regionserver.HStore(1080): Added hdfs://localhost:45625/user/jenkins/test-data/2a140438-d2b1-6994-16d5-13e6c7986734/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/5208485b8bc04aabad9406fd26f01e78, entries=8, sequenceid=66, filesize=6.3 K 2023-02-11 01:00:58,910 INFO [M:0;jenkins-hbase4:46619] regionserver.HRegion(2947): Finished flush of dataSize ~24.07 KB/24646, heapSize ~29.54 KB/30248, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 31ms, sequenceid=66, compaction requested=false 2023-02-11 01:00:58,912 INFO [M:0;jenkins-hbase4:46619] regionserver.HRegion(1837): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-11 01:00:58,912 DEBUG [M:0;jenkins-hbase4:46619] regionserver.HRegion(1557): Region close journal for 1595e783b53d99cd5eef43b6debb2682: Waiting for close lock at 1676077258879Disabling compacts and flushes for region at 1676077258879Disabling writes for close at 1676077258879Obtaining lock to block concurrent updates at 1676077258879Preparing flush snapshotting stores in 1595e783b53d99cd5eef43b6debb2682 at 1676077258879Finished memstore snapshotting master:store,,1.1595e783b53d99cd5eef43b6debb2682., syncing WAL and waiting on mvcc, flushsize=dataSize=24646, getHeapSize=30248, getOffHeapSize=0, getCellsCount=71 at 1676077258879Flushing stores of master:store,,1.1595e783b53d99cd5eef43b6debb2682. at 1676077258880 (+1 ms)Flushing 1595e783b53d99cd5eef43b6debb2682/proc: creating writer at 1676077258881 (+1 ms)Flushing 1595e783b53d99cd5eef43b6debb2682/proc: appending metadata at 1676077258886 (+5 ms)Flushing 1595e783b53d99cd5eef43b6debb2682/proc: closing flushed file at 1676077258886Flushing 1595e783b53d99cd5eef43b6debb2682/proc: reopening flushed file at 1676077258904 (+18 ms)Finished flush of dataSize ~24.07 KB/24646, heapSize ~29.54 KB/30248, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 31ms, sequenceid=66, compaction requested=false at 1676077258910 (+6 ms)Writing region close event to WAL at 1676077258912 (+2 ms)Closed at 1676077258912 2023-02-11 01:00:58,915 WARN [Close-WAL-Writer-0] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(641): complete file /user/jenkins/test-data/2a140438-d2b1-6994-16d5-13e6c7986734/MasterData/WALs/jenkins-hbase4.apache.org,46619,1676077256908/jenkins-hbase4.apache.org%2C46619%2C1676077256908.1676077257135 not finished, retry = 0 2023-02-11 01:00:58,966 DEBUG [Listener at localhost/43805-EventThread] zookeeper.ZKWatcher(600): regionserver:36543-0x10040e53a700001, quorum=127.0.0.1:55753, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-11 01:00:58,966 INFO [RS:0;jenkins-hbase4:36543] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,36543,1676077256955; zookeeper connection closed. 2023-02-11 01:00:58,968 ERROR [Listener at localhost/43805-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@1d4fb67c rejected from java.util.concurrent.ThreadPoolExecutor@2c622340[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 6] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-11 01:00:58,969 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@460b123c] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@460b123c 2023-02-11 01:00:58,969 DEBUG [Listener at localhost/43805-EventThread] zookeeper.ZKWatcher(600): regionserver:36543-0x10040e53a700001, quorum=127.0.0.1:55753, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-11 01:00:58,969 ERROR [Listener at localhost/43805-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@65060617 rejected from java.util.concurrent.ThreadPoolExecutor@2c622340[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 6] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-11 01:00:59,016 INFO [M:0;jenkins-hbase4:46619] flush.MasterFlushTableProcedureManager(83): stop: server shutting down. 2023-02-11 01:00:59,016 INFO [master:store-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-02-11 01:00:59,017 INFO [M:0;jenkins-hbase4:46619] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:46619 2023-02-11 01:00:59,019 DEBUG [M:0;jenkins-hbase4:46619] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase4.apache.org,46619,1676077256908 already deleted, retry=false 2023-02-11 01:00:59,222 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-datanode.properties,hadoop-metrics2.properties 2023-02-11 01:00:59,267 DEBUG [Listener at localhost/43805-EventThread] zookeeper.ZKWatcher(600): regionserver:38923-0x10040e53a700002, quorum=127.0.0.1:55753, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-11 01:00:59,267 INFO [RS:1;jenkins-hbase4:38923] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,38923,1676077256985; zookeeper connection closed. 2023-02-11 01:00:59,267 ERROR [Listener at localhost/43805-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@7f017f75 rejected from java.util.concurrent.ThreadPoolExecutor@603170e7[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 7] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-11 01:00:59,267 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@59ef5da6] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@59ef5da6 2023-02-11 01:00:59,267 DEBUG [Listener at localhost/43805-EventThread] zookeeper.ZKWatcher(600): regionserver:38923-0x10040e53a700002, quorum=127.0.0.1:55753, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-11 01:00:59,268 ERROR [Listener at localhost/43805-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@603d6986 rejected from java.util.concurrent.ThreadPoolExecutor@603170e7[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 7] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-11 01:00:59,268 INFO [Listener at localhost/43805] util.JVMClusterUtil(335): Shutdown of 1 master(s) and 3 regionserver(s) complete 2023-02-11 01:00:59,367 DEBUG [Listener at localhost/43805-EventThread] zookeeper.ZKWatcher(600): master:46619-0x10040e53a700000, quorum=127.0.0.1:55753, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-11 01:00:59,367 INFO [M:0;jenkins-hbase4:46619] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,46619,1676077256908; zookeeper connection closed. 2023-02-11 01:00:59,367 ERROR [Listener at localhost/43805-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@5d2dfe8e rejected from java.util.concurrent.ThreadPoolExecutor@591271f3[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 27] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-11 01:00:59,368 DEBUG [Listener at localhost/43805-EventThread] zookeeper.ZKWatcher(600): master:46619-0x10040e53a700000, quorum=127.0.0.1:55753, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-11 01:00:59,368 ERROR [Listener at localhost/43805-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@1fadcbb6 rejected from java.util.concurrent.ThreadPoolExecutor@591271f3[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 27] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-11 01:00:59,368 WARN [Listener at localhost/43805] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-02-11 01:00:59,380 INFO [Listener at localhost/43805] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-02-11 01:00:59,484 WARN [BP-1379976888-172.31.14.131-1676077256162 heartbeating to localhost/127.0.0.1:45625] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-02-11 01:00:59,484 WARN [BP-1379976888-172.31.14.131-1676077256162 heartbeating to localhost/127.0.0.1:45625] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-1379976888-172.31.14.131-1676077256162 (Datanode Uuid b724d071-d6c0-4206-8e44-df01650df727) service to localhost/127.0.0.1:45625 2023-02-11 01:00:59,485 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a0aedee6-e991-0fa5-31da-b7ab416d8fdd/cluster_f0858fb3-f374-2678-20e0-92155084cac0/dfs/data/data5/current/BP-1379976888-172.31.14.131-1676077256162] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-11 01:00:59,486 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a0aedee6-e991-0fa5-31da-b7ab416d8fdd/cluster_f0858fb3-f374-2678-20e0-92155084cac0/dfs/data/data6/current/BP-1379976888-172.31.14.131-1676077256162] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-11 01:00:59,487 WARN [Listener at localhost/43805] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-02-11 01:00:59,490 INFO [Listener at localhost/43805] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-02-11 01:00:59,594 WARN [BP-1379976888-172.31.14.131-1676077256162 heartbeating to localhost/127.0.0.1:45625] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-02-11 01:00:59,594 WARN [BP-1379976888-172.31.14.131-1676077256162 heartbeating to localhost/127.0.0.1:45625] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-1379976888-172.31.14.131-1676077256162 (Datanode Uuid 4fad92d9-c302-4bf9-98ed-2c4130b68882) service to localhost/127.0.0.1:45625 2023-02-11 01:00:59,594 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a0aedee6-e991-0fa5-31da-b7ab416d8fdd/cluster_f0858fb3-f374-2678-20e0-92155084cac0/dfs/data/data3/current/BP-1379976888-172.31.14.131-1676077256162] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-11 01:00:59,595 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a0aedee6-e991-0fa5-31da-b7ab416d8fdd/cluster_f0858fb3-f374-2678-20e0-92155084cac0/dfs/data/data4/current/BP-1379976888-172.31.14.131-1676077256162] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-11 01:00:59,596 WARN [Listener at localhost/43805] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-02-11 01:00:59,599 INFO [Listener at localhost/43805] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-02-11 01:00:59,703 WARN [BP-1379976888-172.31.14.131-1676077256162 heartbeating to localhost/127.0.0.1:45625] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-02-11 01:00:59,703 WARN [BP-1379976888-172.31.14.131-1676077256162 heartbeating to localhost/127.0.0.1:45625] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-1379976888-172.31.14.131-1676077256162 (Datanode Uuid 0a3f943a-4e83-433c-8383-2ecc4ab4aae9) service to localhost/127.0.0.1:45625 2023-02-11 01:00:59,704 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a0aedee6-e991-0fa5-31da-b7ab416d8fdd/cluster_f0858fb3-f374-2678-20e0-92155084cac0/dfs/data/data1/current/BP-1379976888-172.31.14.131-1676077256162] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-11 01:00:59,704 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a0aedee6-e991-0fa5-31da-b7ab416d8fdd/cluster_f0858fb3-f374-2678-20e0-92155084cac0/dfs/data/data2/current/BP-1379976888-172.31.14.131-1676077256162] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-11 01:00:59,717 INFO [Listener at localhost/43805] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-02-11 01:00:59,830 INFO [Listener at localhost/43805] zookeeper.MiniZooKeeperCluster(344): Shutdown MiniZK cluster with all ZK servers 2023-02-11 01:00:59,855 INFO [Listener at localhost/43805] hbase.HBaseTestingUtility(1293): Minicluster is down 2023-02-11 01:00:59,866 INFO [Listener at localhost/43805] hbase.ResourceChecker(175): after: client.TestAsyncClusterAdminApi2#testShutdown Thread=106 (was 77) Potentially hanging thread: nioEventLoopGroup-12-1 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-9-2 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: IPC Client (762418739) connection to localhost/127.0.0.1:45625 from jenkins.hfs.4 java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1035) org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079) Potentially hanging thread: IPC Client (762418739) connection to localhost/127.0.0.1:45625 from jenkins java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1035) org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079) Potentially hanging thread: nioEventLoopGroup-12-2 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: LeaseRenewer:jenkins@localhost:45625 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.run(LeaseRenewer.java:411) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.access$600(LeaseRenewer.java:76) org.apache.hadoop.hdfs.client.impl.LeaseRenewer$1.run(LeaseRenewer.java:307) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-7-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: LeaseRenewer:jenkins.hfs.5@localhost:45625 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.run(LeaseRenewer.java:411) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.access$600(LeaseRenewer.java:76) org.apache.hadoop.hdfs.client.impl.LeaseRenewer$1.run(LeaseRenewer.java:307) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-5 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Listener at localhost/43805 java.lang.Thread.dumpThreads(Native Method) java.lang.Thread.getAllStackTraces(Thread.java:1615) org.apache.hadoop.hbase.ResourceCheckerJUnitListener$ThreadResourceAnalyzer.getVal(ResourceCheckerJUnitListener.java:49) org.apache.hadoop.hbase.ResourceChecker.fill(ResourceChecker.java:110) org.apache.hadoop.hbase.ResourceChecker.fillEndings(ResourceChecker.java:104) org.apache.hadoop.hbase.ResourceChecker.end(ResourceChecker.java:206) org.apache.hadoop.hbase.ResourceCheckerJUnitListener.end(ResourceCheckerJUnitListener.java:165) org.apache.hadoop.hbase.ResourceCheckerJUnitListener.testFinished(ResourceCheckerJUnitListener.java:185) org.junit.runner.notification.SynchronizedRunListener.testFinished(SynchronizedRunListener.java:87) org.junit.runner.notification.RunNotifier$9.notifyListener(RunNotifier.java:225) org.junit.runner.notification.RunNotifier$SafeNotifier.run(RunNotifier.java:72) org.junit.runner.notification.RunNotifier.fireTestFinished(RunNotifier.java:222) org.junit.internal.runners.model.EachTestNotifier.fireTestFinished(EachTestNotifier.java:38) org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:372) org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103) org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63) org.junit.runners.ParentRunner$4.run(ParentRunner.java:331) org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79) org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329) org.junit.runners.ParentRunner.access$100(ParentRunner.java:66) org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293) org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) org.apache.hadoop.hbase.SystemExitRule$1.evaluate(SystemExitRule.java:39) org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299) org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293) java.util.concurrent.FutureTask.run(FutureTask.java:266) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-10-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: IPC Client (762418739) connection to localhost/127.0.0.1:45625 from jenkins java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1035) org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079) Potentially hanging thread: nioEventLoopGroup-9-3 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-13-3 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Timer for 'DataNode' metrics system java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: RS-EventLoopGroup-9-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: LeaseRenewer:jenkins.hfs.3@localhost:45625 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.run(LeaseRenewer.java:411) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.access$600(LeaseRenewer.java:76) org.apache.hadoop.hdfs.client.impl.LeaseRenewer$1.run(LeaseRenewer.java:307) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-13-1 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-11-2 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-9-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: IPC Client (762418739) connection to localhost/127.0.0.1:45625 from jenkins.hfs.5 java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1035) org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079) Potentially hanging thread: nioEventLoopGroup-9-1 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-7-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-8-1 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-10-2 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-7-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-8-2 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: IPC Client (762418739) connection to localhost/127.0.0.1:45625 from jenkins.hfs.3 java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:1035) org.apache.hadoop.ipc.Client$Connection.run(Client.java:1079) Potentially hanging thread: nioEventLoopGroup-8-3 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-10-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-11-1 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-10-3 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-13-2 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-12-3 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-8-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-4 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-9-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: LeaseRenewer:jenkins.hfs.4@localhost:45625 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.run(LeaseRenewer.java:411) org.apache.hadoop.hdfs.client.impl.LeaseRenewer.access$600(LeaseRenewer.java:76) org.apache.hadoop.hdfs.client.impl.LeaseRenewer$1.run(LeaseRenewer.java:307) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-10-1 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-10-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: nioEventLoopGroup-11-3 java.lang.Thread.sleep(Native Method) io.netty.util.concurrent.SingleThreadEventExecutor.confirmShutdown(SingleThreadEventExecutor.java:790) io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:525) io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) - Thread LEAK? -, OpenFileDescriptor=537 (was 497) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=171 (was 186), ProcessCount=168 (was 168), AvailableMemoryMB=3063 (was 3171)